Skip to content

Update microbenchmark design to reduce flakiness#8300

Merged
igoragoli merged 91 commits into
masterfrom
augusto/fix-microbenchmarks
Apr 8, 2026
Merged

Update microbenchmark design to reduce flakiness#8300
igoragoli merged 91 commits into
masterfrom
augusto/fix-microbenchmarks

Conversation

@igoragoli

@igoragoli igoragoli commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

Summary of changes

Reason for change

APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark scenarios

Implementation details

Benchmark design changes (what reduced flakiness)

  1. Run every benchmark 10 times (launchCount).
    • This is important to mitigate inter-run variance, which can be caused by things such as varying memory placement across runs.
  2. Manually set warmup iterations to 10 (warmupCount).
    • On average, the number of automatically determined warmup iterations is ~15 for the current benchmarking setup.
    • I got this down to 10 because:
      • I needed to make benchmarks faster.
      • On average, every iteration runs the benchmark method for hundreds of thousands of times, so I hypothesized that 5 less warmup iterations would not affect results significantly.
      • We were already doing 10 runs. I hypothesized that stability would come from different process samples, not from a deeper warmup within a single process.
  3. Manually set actual iterations to 10 (iterationCount).
    • On average, the number of automatically determined warmup iterations is ~35 for the current benchmarking setup. Some (CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces, StringConcatAspectBenchmark, StringConcatBenchmark) go up to 120.
    • I got this down to 10 because of the same reasons described on change 2.
  4. Kept a "base" iterationTime of 200 ms, but set iterationTime on flaky benchmarks to 500 ms.
    • Increasing the iterationTime for flaky benchmarks helps immensely. We're able to capture much more variation into our samples once we fit more operations into the iteration. I tried many configurations, this turned out to be the minimum we need.

Benchmark execution changes (what allowed me to implement the benchmark design changes)

  1. Moved code from benchmarking-platform @ dd-trace-dotnet/micro to here.
    • This is just easier to understand, and doesn't require changing two repos to update benchmarks.
    • ⚠️ We just have to move the ami.yaml to build the AMIs with which we run microbenchmarks. But there's some preliminary work to avoid dd-octo-sts failures on this repo. Captured on APMSP-2736.
  2. Used bp-runner to run the benchmarks, since it allows us to run microbenchmarks in parallel on isolated CPU sets.
    • This was crucial to allow us to run 10 launches. The current benchmarking job takes 50 minutes (20 min build + 30 min benchmarks). By running things 10 times sequentially it would take us 320 minutes.
  3. Split benchmarks into two batches: flaky ones and non-flaky ones. I moved the non-flaky ones to CPUs in NUMA node 1 because Windows often runs system processes on the first NUMA node, which can affect results.

Benchmark class/Build.cs changes

  1. Build.cs:
    • Added a BuildBenchmarks target to build all benchmark executables in one go (needed for bp-runner parallel execution).
  2. AppSecWafBenchmark.cs / AppSecBenchmarkUtils.cs: Fixed path resolution to make .NET Framework 4.7.2 benchmarks run in parallel.
    • Changed Directory.GetCurrentDirectory() and Environment.CurrentDirectory to AppContext.BaseDirectory.
    • BenchmarkDotNet spawns a separate process for each benchmark run, and on .NET Framework the working directory doesn't match the executable directory. This caused WAF initialization to fail with "No valid rules found" because the rules file and native libraries couldn't be located.
  • Added [IterationTime(500)] attribute to benchmarks identified as flaky (e.g., AspNetCoreBenchmark).

Test coverage

Other details

How did we measure flakiness?

  1. Run the run-benchmarks job for the same commit 10 times.
  2. Cross-validate results.
    • For every benchmark and metric (execution_time, throughtput, and allocated_mem✨), we compute the false positive rate and the max confidence interval difference.
    • The false positive rate is how often the benchmark system incorrectly flags a "regression" when comparing identical code.
    • The max confidence interval difference is the largest spurious difference observed across all 45 comparisons of run-benchmarks results when comparing identical code.

How much did flakiness drop?

  • The false positive rate dropped from 5.7% to 3.8%.
  • The maximum CI difference dropped from 56% to 16.66%.

See the per-benchmark/metric breakdown

This graph shows intuitively the distribution of FP rates and max CI differences for all comparisons. The more weight the distribution has closer to 0, the better. Memory allocation results are already pretty stable, so it's hard to see a clear win there.

image

How were flaky benchmarks determined (necessary to choose a higher iterationTime for them)?

  1. Started with existing ignored list @NachoEchevarria and I created.
  2. Ran stability experiments and refined based on data (again, per-benchmark/metric breakdown)
    • AppSec benchmarks (AppSecWafBenchmark, AppSecBodyBenchmark): Moved to stable batch. With the new design, most FP rates are under 5%, only one outlier at 20% for the netcoreapp3.1 runtime.
    • AspNetCoreBenchmark: Consistently over 20% FP even with optimized design.
    • AgentWriterBenchmark: Added to flaky batch. Most methods are stable, but WriteAndFlushEnrichedTraces netcoreapp3.1 is VERY problematic.

Phase 1 of modernizing dd-trace-dotnet microbenchmarks:
- Add bp-runner.windows.yml with single dry-run benchmark
- Add instance.yml provision file for Windows instances
- Add run-benchmarks-test job using bp-infra with --env-regex
- Uses test AMI with bp-runner from dev branch

The new job runs automatically on augusto/fix-microbenchmarks branch
and manually on other branches. It validates the new infrastructure
by running a single SpanBenchmark.StartFinishSpan with --job dry.
Replace the shell-script-based benchmark execution with bp-infra and
bp-runner. This uses the instance.yml provision file and
bp-runner.windows.yml config directly from dd-trace-dotnet repo.
bp-runner passes $CodeSrc and $BaselineOrCandidate as function params,
not environment variables.
@dd-trace-dotnet-ci-bot

dd-trace-dotnet-ci-bot Bot commented Mar 13, 2026

Copy link
Copy Markdown

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8300) and master.

⚠️ Potential regressions detected

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration189.63 ± (189.56 - 190.28) ms211.55 ± (211.17 - 212.10) ms+11.6%❌⬆️
.NET Framework 4.8 - Bailout
duration193.19 ± (193.12 - 193.51) ms216.95 ± (216.62 - 217.64) ms+12.3%❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1136.69 ± (1141.07 - 1150.90) ms1236.93 ± (1234.11 - 1241.31) ms+8.8%❌⬆️
Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration72.29 ± (72.25 - 72.61) ms76.57 ± (76.52 - 76.96) ms+5.9%✅⬆️
.NET Framework 4.8 - Bailout
duration76.69 ± (76.48 - 76.84) ms80.52 ± (80.29 - 80.69) ms+5.0%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1062.49 ± (1063.55 - 1069.14) ms1110.69 ± (1109.77 - 1115.09) ms+4.5%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.40 ± (22.36 - 22.44) ms23.11 ± (23.05 - 23.16) ms+3.2%✅⬆️
process.time_to_main_ms83.81 ± (83.62 - 84.00) ms88.90 ± (88.71 - 89.09) ms+6.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.89 ± (10.89 - 10.90) MB10.92 ± (10.92 - 10.93) MB+0.3%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%
.NET Core 3.1 - Bailout
process.internal_duration_ms22.31 ± (22.27 - 22.35) ms22.90 ± (22.85 - 22.96) ms+2.7%✅⬆️
process.time_to_main_ms84.90 ± (84.70 - 85.10) ms89.80 ± (89.59 - 90.01) ms+5.8%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.87 ± (10.86 - 10.87) MB10.95 ± (10.95 - 10.96) MB+0.8%✅⬆️
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms224.53 ± (223.23 - 225.84) ms232.73 ± (231.53 - 233.92) ms+3.7%✅⬆️
process.time_to_main_ms515.80 ± (514.61 - 516.98) ms544.20 ± (542.79 - 545.60) ms+5.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed47.88 ± (47.84 - 47.92) MB47.83 ± (47.80 - 47.86) MB-0.1%
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+1.3%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms20.91 ± (20.87 - 20.94) ms21.86 ± (21.81 - 21.91) ms+4.6%✅⬆️
process.time_to_main_ms71.75 ± (71.60 - 71.89) ms77.23 ± (77.05 - 77.42) ms+7.6%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.61 ± (10.60 - 10.61) MB10.64 ± (10.64 - 10.65) MB+0.4%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 6 - Bailout
process.internal_duration_ms20.95 ± (20.91 - 20.98) ms21.70 ± (21.64 - 21.76) ms+3.6%✅⬆️
process.time_to_main_ms72.97 ± (72.79 - 73.15) ms78.21 ± (78.03 - 78.38) ms+7.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed10.70 ± (10.70 - 10.71) MB10.76 ± (10.76 - 10.77) MB+0.6%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms389.19 ± (387.12 - 391.26) ms385.79 ± (383.66 - 387.91) ms-0.9%
process.time_to_main_ms515.46 ± (514.56 - 516.36) ms545.94 ± (544.71 - 547.17) ms+5.9%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed49.84 ± (49.81 - 49.87) MB49.86 ± (49.83 - 49.89) MB+0.0%✅⬆️
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)+0.1%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms19.26 ± (19.23 - 19.29) ms20.24 ± (20.19 - 20.28) ms+5.1%✅⬆️
process.time_to_main_ms71.85 ± (71.67 - 72.03) ms77.00 ± (76.83 - 77.18) ms+7.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.68 ± (7.68 - 7.69) MB7.68 ± (7.68 - 7.69) MB-0.0%
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%
.NET 8 - Bailout
process.internal_duration_ms19.12 ± (19.08 - 19.15) ms20.17 ± (20.12 - 20.23) ms+5.5%✅⬆️
process.time_to_main_ms72.23 ± (72.10 - 72.37) ms77.63 ± (77.45 - 77.81) ms+7.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed7.73 ± (7.72 - 7.73) MB7.74 ± (7.74 - 7.75) MB+0.2%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms306.86 ± (304.63 - 309.09) ms306.54 ± (304.14 - 308.94) ms-0.1%
process.time_to_main_ms477.29 ± (476.54 - 478.04) ms505.37 ± (504.62 - 506.12) ms+5.9%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%
runtime.dotnet.mem.committed37.03 ± (37.00 - 37.05) MB37.08 ± (37.05 - 37.12) MB+0.2%✅⬆️
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)-1.0%

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration189.63 ± (189.56 - 190.28) ms211.55 ± (211.17 - 212.10) ms+11.6%❌⬆️
.NET Framework 4.8 - Bailout
duration193.19 ± (193.12 - 193.51) ms216.95 ± (216.62 - 217.64) ms+12.3%❌⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1136.69 ± (1141.07 - 1150.90) ms1236.93 ± (1234.11 - 1241.31) ms+8.8%❌⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms184.99 ± (184.71 - 185.26) ms207.92 ± (207.44 - 208.40) ms+12.4%✅⬆️
process.time_to_main_ms79.28 ± (79.10 - 79.46) ms90.60 ± (90.34 - 90.85) ms+14.3%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed16.07 ± (16.04 - 16.09) MB15.97 ± (15.95 - 15.98) MB-0.6%
runtime.dotnet.threads.count20 ± (19 - 20)20 ± (20 - 20)+1.8%✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms184.18 ± (183.98 - 184.39) ms209.90 ± (209.39 - 210.42) ms+14.0%✅⬆️
process.time_to_main_ms80.62 ± (80.54 - 80.70) ms93.03 ± (92.76 - 93.29) ms+15.4%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed15.99 ± (15.86 - 16.12) MB15.94 ± (15.92 - 15.96) MB-0.3%
runtime.dotnet.threads.count20 ± (20 - 20)21 ± (21 - 21)+5.0%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms388.46 ± (387.03 - 389.89) ms419.25 ± (417.54 - 420.95) ms+7.9%✅⬆️
process.time_to_main_ms501.31 ± (500.27 - 502.35) ms550.14 ± (548.73 - 551.55) ms+9.7%✅⬆️
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%
runtime.dotnet.mem.committed57.82 ± (57.61 - 58.03) MB58.77 ± (58.74 - 58.81) MB+1.6%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+1.4%✅⬆️
.NET 6 - Baseline
process.internal_duration_ms189.48 ± (189.23 - 189.74) ms215.71 ± (215.20 - 216.22) ms+13.8%✅⬆️
process.time_to_main_ms69.07 ± (68.96 - 69.18) ms80.50 ± (80.22 - 80.77) ms+16.5%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed15.84 ± (15.67 - 16.01) MB16.19 ± (16.17 - 16.21) MB+2.2%✅⬆️
runtime.dotnet.threads.count18 ± (18 - 18)20 ± (19 - 20)+8.5%✅⬆️
.NET 6 - Bailout
process.internal_duration_ms188.90 ± (188.71 - 189.09) ms215.84 ± (215.25 - 216.43) ms+14.3%✅⬆️
process.time_to_main_ms70.06 ± (70.00 - 70.12) ms81.73 ± (81.46 - 82.00) ms+16.7%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed15.71 ± (15.54 - 15.88) MB16.23 ± (16.21 - 16.25) MB+3.3%✅⬆️
runtime.dotnet.threads.count19 ± (18 - 19)21 ± (21 - 21)+11.2%✅⬆️
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms601.33 ± (598.64 - 604.02) ms600.70 ± (598.24 - 603.17) ms-0.1%
process.time_to_main_ms502.36 ± (501.71 - 503.00) ms557.24 ± (556.10 - 558.38) ms+10.9%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed61.55 ± (61.45 - 61.64) MB61.24 ± (61.14 - 61.34) MB-0.5%
runtime.dotnet.threads.count30 ± (30 - 30)31 ± (31 - 31)+1.8%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms187.22 ± (186.97 - 187.48) ms213.71 ± (213.14 - 214.29) ms+14.1%✅⬆️
process.time_to_main_ms68.72 ± (68.55 - 68.89) ms79.41 ± (79.13 - 79.68) ms+15.6%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.85 ± (11.82 - 11.88) MB11.60 ± (11.58 - 11.61) MB-2.2%
runtime.dotnet.threads.count18 ± (18 - 18)19 ± (19 - 19)+4.8%✅⬆️
.NET 8 - Bailout
process.internal_duration_ms186.28 ± (186.10 - 186.47) ms212.61 ± (212.13 - 213.08) ms+14.1%✅⬆️
process.time_to_main_ms69.44 ± (69.38 - 69.51) ms80.62 ± (80.34 - 80.90) ms+16.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed11.69 ± (11.58 - 11.79) MB11.68 ± (11.65 - 11.70) MB-0.1%
runtime.dotnet.threads.count18 ± (18 - 19)20 ± (20 - 20)+9.2%✅⬆️
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms521.37 ± (518.81 - 523.93) ms548.64 ± (541.53 - 555.74) ms+5.2%✅⬆️
process.time_to_main_ms461.52 ± (460.83 - 462.20) ms512.63 ± (511.56 - 513.71) ms+11.1%✅⬆️
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%
runtime.dotnet.mem.committed50.69 ± (50.67 - 50.72) MB51.01 ± (50.92 - 51.10) MB+0.6%✅⬆️
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)+1.3%✅⬆️
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8)
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (77ms)  : 74, 80
    master - mean (72ms)  : 70, 75

    section Bailout
    This PR (8300) - mean (80ms)  : 78, 83
    master - mean (77ms)  : 75, 78

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (1,112ms)  : 1074, 1151
    master - mean (1,066ms)  : 1027, 1106

Loading
FakeDbCommand (.NET Core 3.1)
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (119ms)  : 116, 122
    master - mean (113ms)  : 110, 116

    section Bailout
    This PR (8300) - mean (120ms)  : 117, 123
    master - mean (114ms)  : 111, 117

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (815ms)  : 794, 837
    master - mean (778ms)  : 755, 801

Loading
FakeDbCommand (.NET 6)
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (106ms)  : 102, 110
    master - mean (99ms)  : 96, 102

    section Bailout
    This PR (8300) - mean (107ms)  : crit, 103, 110
    master - mean (100ms)  : 97, 102

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (960ms)  : 927, 993
    master - mean (933ms)  : 900, 966

Loading
FakeDbCommand (.NET 8)
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (105ms)  : 102, 109
    master - mean (99ms)  : 95, 102

    section Bailout
    This PR (8300) - mean (106ms)  : crit, 103, 109
    master - mean (99ms)  : 97, 101

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (843ms)  : 806, 879
    master - mean (814ms)  : 778, 850

Loading
HttpMessageHandler (.NET Framework 4.8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (212ms)  : 207, 217
    master - mean (190ms)  : 186, 193

    section Bailout
    This PR (8300) - mean (217ms)  : crit, 212, 223
    master - mean (193ms)  : 191, 195

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (1,238ms)  : crit, 1185, 1290
    master - mean (1,146ms)  : 1072, 1220

Loading
HttpMessageHandler (.NET Core 3.1)
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (308ms)  : 298, 318
    master - mean (272ms)  : 268, 276

    section Bailout
    This PR (8300) - mean (313ms)  : crit, 303, 323
    master - mean (272ms)  : 270, 275

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (1,012ms)  : crit, 981, 1042
    master - mean (919ms)  : 892, 946

Loading
HttpMessageHandler (.NET 6)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (305ms)  : 296, 315
    master - mean (266ms)  : 262, 271

    section Bailout
    This PR (8300) - mean (307ms)  : crit, 300, 313
    master - mean (267ms)  : 264, 269

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (1,191ms)  : 1144, 1238
    master - mean (1,133ms)  : 1093, 1173

Loading
HttpMessageHandler (.NET 8)
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8300) - mean (304ms)  : 297, 311
    master - mean (265ms)  : 262, 269

    section Bailout
    This PR (8300) - mean (304ms)  : crit, 298, 310
    master - mean (265ms)  : 262, 268

    section CallTarget+Inlining+NGEN
    This PR (8300) - mean (1,101ms)  : crit, 989, 1212
    master - mean (1,016ms)  : 966, 1066

Loading

The .NET SDK is installed at C:\dotnet on the AMI, but it's not in PATH
by default for the bp-runner execution context.
The test AMI has .NET 3.1, 6.0, and 10.0 installed but not 8.0.
Change the benchmark target framework to net6.0.
Add Phase 2.5: S3 artifact persistence for microbenchmarks
- Add upload_results provision step in instance.yml using Write-S3Object
- Add S3 fetch step in microbenchmarks.yml after bp-infra launch
- Update _latest on master branch for baseline comparison
Output format changes from:
  candidate-Benchmarks.Trace.SpanBenchmark-report-full-compressed-0.json
To:
  candidate.Trace.SpanBenchmark.json

Strips 'Benchmarks.' prefix and '-report(-full)?(-compressed)?' suffix.
When running parallel benchmarks, multiple dotnet run commands were
trying to rebuild simultaneously, causing build failures. Adding
--no-build uses the pre-built output from how_to_fetch_release.
The --no-build flag was failing because only the tracer was being built,
not the benchmarks project itself. Now we build both in how_to_fetch_release
so parallel runs can use --no-build successfully.
- Add GetBenchmarkProjectsWithSettings() to share project list between targets
- Add independent BuildBenchmarks target to pre-build benchmark projects
- Add NoBenchmarkBuild parameter to skip building in RunBenchmarks
- Add BenchmarkArtifactsDirectory parameter for configurable output
- Make EnsureCleanDirectory conditional (only in sequential mode)
- Update bp-runner.windows.yml to use Nuke build targets:
  - how_to_fetch_release: build.cmd BuildTracerHome BuildBenchmarks
  - how_to_run_benchmarks: build.cmd RunBenchmarks --no-benchmark-build
Parallel bp-runner invocations race to compile/load Nuke's _build.dll,
causing file lock errors. Add retry with 3 attempts and 5s delay.
Run benchmark executables directly instead of via Nuke to avoid
_build.dll lock contention during parallel bp-runner invocations.

Add analysis scripts (fetch-results, analyze-results, post-pr-comment,
upload-to-bp-ui) copied from benchmarking-platform and simplified.
- Fix BenchmarkDotNet runtime arguments (array instead of string)
- Use correct runtime names (netcoreapp31, net60 instead of dotted)
- Add fetch/analyze/upload steps to run-benchmarks job
Run `dotnet build/_build/bin/Debug/_build.dll RunBenchmarks` instead of
`build.cmd` to avoid _build.dll file lock when parallel bp-runner
invocations try to recompile Nuke's build project simultaneously.
…tion

- bp-runner now calls run-benchmarks.ps1 instead of Nuke DLL
- Removes runtime (-r) argument, uses BenchmarkDotNet defaults
- Each parallel run gets unique artifacts directory via PARALLEL_INDEX
- Bypasses Nuke entirely to avoid .nuke/build.schema.json file locks
Without -r flag, BenchmarkDotNet fails to generate benchmark harnesses
properly. Add back the runtime targets (net472, netcoreapp3.1, net6.0)
to match Build.cs RunBenchmarks behavior.
@pr-commenter

pr-commenter Bot commented Mar 20, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-04-07 13:35:03

Comparing candidate commit 2ab96a5 in PR branch augusto/fix-microbenchmarks with baseline commit 84efe2b in branch master.

Found 32 performance improvements and 38 performance regressions! Performance is the same for 214 metrics, 4 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0

  • 🟩 throughput [+9051.861op/s; +10931.868op/s] or [+7.706%; +9.306%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 execution_time [+290.360ms; +292.585ms] or [+129.357%; +130.348%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+380.395ms; +382.490ms] or [+308.568%; +310.267%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+394.955ms; +398.664ms] or [+348.833%; +352.109%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

  • 🟩 throughput [+6410.811op/s; +8544.783op/s] or [+5.806%; +7.739%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

  • 🟩 throughput [+108100.885op/s; +118746.604op/s] or [+11.833%; +12.998%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟩 throughput [+98219.114op/s; +114777.762op/s] or [+14.079%; +16.452%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0

  • 🟩 execution_time [-18.171ms; -14.824ms] or [-8.473%; -6.913%]
  • 🟩 throughput [+9702.141op/s; +12367.327op/s] or [+6.121%; +7.802%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1

  • 🟩 throughput [+7372.247op/s; +9789.412op/s] or [+5.756%; +7.643%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • 🟩 throughput [+246389.329op/s; +332046.703op/s] or [+7.930%; +10.687%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 throughput [+186015.612op/s; +245637.888op/s] or [+7.313%; +9.656%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

  • 🟥 execution_time [+300.625ms; +301.283ms] or [+150.325%; +150.654%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

  • 🟥 execution_time [+298.965ms; +300.705ms] or [+149.324%; +150.193%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • 🟥 execution_time [+294.221ms; +296.753ms] or [+143.966%; +145.205%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

  • 🟥 execution_time [+298.003ms; +298.979ms] or [+147.111%; +147.593%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 execution_time [+296.274ms; +299.277ms] or [+145.650%; +147.127%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+299.907ms; +301.302ms] or [+148.814%; +149.506%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0

  • 🟥 execution_time [+24.033µs; +35.859µs] or [+5.506%; +8.215%]
  • 🟥 throughput [-177.846op/s; -118.456op/s] or [-7.759%; -5.168%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

  • 🟥 execution_time [+29.596µs; +48.366µs] or [+9.568%; +15.635%]
  • 🟥 throughput [-453.002op/s; -286.019op/s] or [-13.974%; -8.823%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

  • 🟥 execution_time [+300.743ms; +301.531ms] or [+150.860%; +151.255%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • 🟥 execution_time [+300.274ms; +305.737ms] or [+149.461%; +152.180%]
  • 🟩 throughput [+1256.160op/s; +1373.074op/s] or [+10.377%; +11.343%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

  • 🟩 throughput [+659.500op/s; +812.870op/s] or [+6.371%; +7.852%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 throughput [-567.182op/s; -522.171op/s] or [-47.548%; -43.775%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+326.300ms; +331.021ms] or [+188.792%; +191.523%]
  • 🟥 throughput [-701.309op/s; -665.934op/s] or [-47.243%; -44.860%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 throughput [-343.308op/s; -304.612op/s] or [-25.234%; -22.390%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0

  • 🟩 execution_time [-196.385µs; -183.680µs] or [-16.094%; -15.053%]
  • 🟩 throughput [+146.453op/s; +156.924op/s] or [+17.870%; +19.148%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472

  • 🟩 execution_time [-195.172µs; -174.903µs] or [-6.944%; -6.223%]
  • 🟩 throughput [+23.763op/s; +26.668op/s] or [+6.678%; +7.495%]

scenario:Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net6.0

  • 🟩 throughput [+39253.568op/s; +45692.723op/s] or [+7.614%; +8.863%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

  • 🟥 execution_time [+303.467ms; +305.158ms] or [+153.662%; +154.518%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • 🟥 execution_time [+298.218ms; +300.369ms] or [+146.650%; +147.708%]
  • 🟩 throughput [+64608.153op/s; +72052.662op/s] or [+10.699%; +11.931%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

  • 🟥 execution_time [+304.174ms; +307.235ms] or [+154.990%; +156.550%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • 🟥 execution_time [+297.478ms; +299.223ms] or [+148.645%; +149.517%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟥 execution_time [+316.149ms; +320.406ms] or [+172.155%; +174.473%]
  • 🟩 throughput [+43333.026op/s; +56292.425op/s] or [+7.252%; +9.421%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 execution_time [+297.517ms; +300.951ms] or [+146.384%; +148.074%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

  • 🟥 execution_time [+304.157ms; +305.908ms] or [+153.961%; +154.847%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • 🟥 execution_time [+302.499ms; +306.986ms] or [+155.056%; +157.356%]
  • 🟩 throughput [+28609.816op/s; +40398.933op/s] or [+5.408%; +7.637%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • 🟥 execution_time [+303.396ms; +306.857ms] or [+154.291%; +156.052%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1

  • 🟩 throughput [+15638.249op/s; +17991.276op/s] or [+5.792%; +6.664%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟩 allocated_mem [-19.948KB; -19.924KB] or [-7.279%; -7.270%]
  • 🟩 throughput [+118.045op/s; +277.175op/s] or [+6.288%; +14.764%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

  • 🟥 throughput [-3407.943op/s; -1325.540op/s] or [-15.640%; -6.083%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

  • 🟥 execution_time [+296.704ms; +298.104ms] or [+147.211%; +147.905%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+307.058ms; +310.071ms] or [+160.576%; +162.151%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+305.889ms; +307.559ms] or [+157.399%; +158.258%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

  • 🟩 throughput [+39809.929op/s; +51502.189op/s] or [+7.431%; +9.613%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

  • 🟥 execution_time [+302.236ms; +303.657ms] or [+151.415%; +152.127%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+299.117ms; +302.686ms] or [+148.177%; +149.945%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+300.890ms; +303.236ms] or [+150.594%; +151.769%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

  • 🟥 execution_time [+300.132ms; +301.365ms] or [+150.258%; +150.876%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • 🟥 execution_time [+409.367ms; +412.675ms] or [+440.635%; +444.196%]
  • 🟩 throughput [+1276.240op/s; +1514.777op/s] or [+10.011%; +11.883%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟥 execution_time [+300.863ms; +301.712ms] or [+150.861%; +151.287%]
  • 🟩 throughput [+18461881.128op/s; +19163496.864op/s] or [+8.185%; +8.496%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • 🟩 throughput [+65377.715op/s; +76040.264op/s] or [+5.963%; +6.935%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1

  • 🟩 execution_time [-15.412ms; -11.087ms] or [-7.231%; -5.202%]
  • 🟩 throughput [+51439.930op/s; +69690.393op/s] or [+6.004%; +8.135%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • 🟩 throughput [+72028.279op/s; +100993.169op/s] or [+5.489%; +7.696%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

  • 🟩 throughput [+35767.401op/s; +47928.555op/s] or [+6.294%; +8.433%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1

  • 🟩 throughput [+26147.489op/s; +28896.275op/s] or [+5.833%; +6.446%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • 🟩 throughput [+67149.335op/s; +76329.293op/s] or [+7.382%; +8.391%]

Use AppContext.BaseDirectory instead of Directory.GetCurrentDirectory()
for locating benchmark files. On .NET Framework 4.7.2, when BenchmarkDotNet
spawns a separate process, the working directory may differ from the exe
location, causing WAF rules to not be found.

Also removes BenchmarkCategory from AgentWriterBenchmark to exclude
WriteEnrichedTraces from PR/master benchmark runs.
"-m",
"-f", $Filter,
"--allCategories", $Category,
"--iterationTime", "200", # Increased to 500 ms for flaky benchmarks

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line seems kind of contradictory

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in ee54ed7.

Comment thread .gitlab/benchmarks/microbenchmarks/scripts/run-benchmarks.ps1
@NachoEchevarria

Copy link
Copy Markdown
Collaborator

There are several files that should not be commited:

  • after.csv / before.csv
  • instance.rdp
  • plan.md
  • prompt.md
  • raw
  • summary.md
  • tmp.ps1


Write-Output "=== Benchmarks completed for filter: $env:PARALLEL_ITEM ==="

.how_to_run_benchmarks_batch1: &how_to_run_benchmarks_batch1 |

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is nearly identical to how_to_run_benchmarks_batch0. They could probably be refactored, something like:

experiments:
- name: batch-0
steps:
- name: batch-0
environment:
BATCH_PREFIX: "b0"
how_to_run_benchmarks: *how_to_run_benchmarks
- name: batch-1
steps:
- name: batch-1
environment:
BATCH_PREFIX: "b1"
how_to_run_benchmarks: *how_to_run_benchmarks

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in da7b7b6.

@igoragoli

igoragoli commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

Here is a comparison of memory allocation measurements using these benchmarks for 10c5701 and 10c5701^:

That PR added 8 bytes. And the median byte delta, when positive, is +8, as expected.

image
Click to show table.
Benchmark Scenario 10c5701^ (bytes) 10c5701 (bytes) Δ (bytes) Δ (%)
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0 253972.74 248764.33 -5208.41 -2.05%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0 520.00 528.00 +8.00 +1.54%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1 520.00 528.00 +8.00 +1.54%
Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472 521.99 530.00 +8.00 +1.53%
Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472 1075.02 1090.99 +15.97 +1.49%
Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0 1192.01 1207.99 +15.99 +1.34%
Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1 1192.01 1207.99 +15.99 +1.34%
Benchmarks.Trace.SpanBenchmark.StartFinishScope net472 602.00 610.00 +8.01 +1.33%
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472 602.00 610.00 +8.01 +1.33%
Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0 640.01 648.00 +7.99 +1.25%
Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1 640.01 648.00 +7.99 +1.25%
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0 640.01 648.00 +7.99 +1.25%
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1 640.01 648.00 +7.99 +1.25%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0 759.99 768.00 +8.01 +1.05%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0 783.99 792.01 +8.02 +1.02%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1 783.99 792.01 +8.02 +1.02%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472 794.00 802.01 +8.01 +1.01%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0 1584.01 1599.98 +15.98 +1.01%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472 1599.98 1616.00 +16.01 +1.00%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1 1632.00 1648.00 +16.00 +0.98%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1 832.01 840.00 +7.99 +0.96%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0 1688.00 1704.01 +16.00 +0.95%
Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1 1696.00 1711.98 +15.98 +0.94%
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472 851.00 859.00 +8.00 +0.94%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472 859.00 866.99 +7.99 +0.93%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0 896.01 903.98 +7.98 +0.89%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1 896.01 903.98 +7.98 +0.89%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net472 930.99 939.00 +8.01 +0.86%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery netcoreapp3.1 959.99 968.00 +8.00 +0.83%
Benchmarks.Trace.DbCommandBenchmark.ExecuteNonQuery net6.0 968.00 976.01 +8.01 +0.83%
Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472 2026.01 2041.99 +15.98 +0.79%
Benchmarks.Trace.NLogBenchmark.EnrichedLog net472 2049.03 2064.97 +15.94 +0.78%
Benchmarks.Trace.RedisBenchmark.SendReceive net6.0 1032.00 1040.00 +8.00 +0.78%
Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1 1032.00 1040.00 +8.00 +0.78%
Benchmarks.Trace.RedisBenchmark.SendReceive net472 1035.00 1042.99 +7.99 +0.77%
Benchmarks.Trace.NLogBenchmark.EnrichedLog net6.0 2244.01 2259.98 +15.97 +0.71%
Benchmarks.Trace.NLogBenchmark.EnrichedLog netcoreapp3.1 2256.04 2272.00 +15.97 +0.71%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1 1200.00 1207.99 +7.99 +0.67%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0 1215.97 1224.02 +8.04 +0.66%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472 1228.00 1236.00 +7.99 +0.65%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0 3264.00 3279.96 +15.96 +0.49%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1 3456.06 3471.98 +15.93 +0.46%
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472 3883.00 3900.02 +17.02 +0.44%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1 274971.00 276154.56 +1183.56 +0.43%
Benchmarks.Trace.HttpClientBenchmark.SendAsync net6.0 2111.99 2119.99 +8.00 +0.38%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472 4510.99 4527.98 +16.99 +0.38%
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472 3287.94 3299.97 +12.04 +0.37%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0 4312.05 4327.06 +15.02 +0.35%
Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1 4312.05 4327.06 +15.02 +0.35%
Benchmarks.Trace.HttpClientBenchmark.SendAsync netcoreapp3.1 2632.03 2640.04 +8.02 +0.30%
Benchmarks.Trace.HttpClientBenchmark.SendAsync net472 2902.99 2910.99 +8.00 +0.28%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1 4616.01 4624.05 +8.04 +0.17%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472 4741.99 4750.07 +8.08 +0.17%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0 4720.09 4728.12 +8.03 +0.17%
Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0 17191.94 17216.12 +24.18 +0.14%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472 59036.46 59109.18 +72.73 +0.12%
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1 20821.97 20846.45 +24.47 +0.12%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1 42582.86 42537.51 -45.35 -0.11%
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0 18015.30 18004.85 -10.44 -0.06%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0 41125.71 41127.74 +2.02 +0.00%
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0 2696.03 2696.03 0.00 0.00%
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1 2696.03 2696.03 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0 280.00 280.00 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1 272.00 272.00 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472 281.00 281.00 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0 3783.98 3783.98 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1 3687.99 3687.99 0.00 0.00%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472 3795.06 3795.06 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0 32408.15 32408.15 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1 32408.15 32408.15 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472 32510.92 32510.92 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0 2143.97 2143.97 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1 2143.97 2143.97 0.00 0.00%
Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472 2146.99 2146.99 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0 4551.98 4551.98 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1 4480.01 4480.01 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472 0.00 0.00 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0 2240.03 2240.03 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1 2216.00 2216.00 0.00 0.00%
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472 0.00 0.00 0.00 0.00%
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0 639997.34 639997.34 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1 640047.25 640047.25 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472 638965.37 638965.37 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1 0.00 0.00 0.00 0.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472 0.00 0.00 0.00 0.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0 42511.66 42511.66 0.00 0.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1 42536.29 42536.29 0.00 0.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472 49152.24 49152.24 0.00 0.00%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472 270333.18 270333.18 0.00 0.00%
Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1 0.00 0.00 0.00 0.00%
Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472 0.00 0.00 0.00 0.00%

@igoragoli
igoragoli requested a review from andrewlock April 1, 2026 11:26

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mostly looked at the Nuke changes, I take your word for it on the iteration changes etc, and 🙈 on the gitlab yaml and scripts 😅

FWIW, ideally we wouldn't have any of those scripts. That's why we have Nuke, to avoid having bash in YAML 😉 But we can always convert them later anyway

Comment thread tracer/build/_build/Build.cs Outdated
Comment thread tracer/test/benchmarks/Benchmarks.Trace/Asm/AppSecBenchmarkUtils.cs

@NachoEchevarria NachoEchevarria left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@igoragoli

igoragoli commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

@andrewlock, just to be extra extra extra sure, I ran stability tests again after these last changes (ran 10 separate run-benchmarks jobs, cross-validated all their results).

The false positive rate was 2.1%, which is consistent with our previous results.

Addressing some remaining comments:

ideally we wouldn't have any of those scripts

Absolutely. If we could implement everything in Nuke and BenchmarkDotNet alone, I would.

But the design fixes here require running benchmarks in parallel, otherwise CI times get even longer. Parallel execution requires bp-runner, which means using scripts.

These scripts originated from the benchmarking-platform repo. @NachoEchevarria and I decided to move them here so there's no switching between repos to understand or debug the pipeline.

We can move them back if you both prefer, but having everything in one place has been much easier to work with.

I take your word for it on the iteration changes etc, and 🙈 on the gitlab yaml and scripts 😅

There's no need to take my word for it! We've ran extensive tests to validate everything is working as expected 😅

Once again, thx for the review!

@igoragoli
igoragoli merged commit d5d94dd into master Apr 8, 2026
136 of 138 checks passed
@igoragoli
igoragoli deleted the augusto/fix-microbenchmarks branch April 8, 2026 04:58
@github-actions github-actions Bot added this to the vNext-v3 milestone Apr 8, 2026
igoragoli added a commit that referenced this pull request Apr 8, 2026
commit d5d94dd7eb8e5ec080ffbe3b5b7d89aafe95fef9
Author: Augusto de Oliveira <[email protected]>
Date:   Wed Apr 8 06:58:34 2026 +0200

    Update microbenchmark design to reduce flakiness (#8300)

    ## Summary of changes

    - Update the benchmark design to reduce flakiness.
    - See [other details for flakiness reduction results](#other-details).
    - Move benchmarking from [benchmarking-platform @
    dd-trace-dotnet/micro](https://github.com/DataDog/benchmarking-platform/tree/dd-trace-dotnet/micro)
    to this repo.
        - This is why we have +1k lines (sorry!).
    - Include `BytesAllocatedPerOperation` as `allocated_mem` into our
    benchmarking results.

    ## Reason for change

    [APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark
    scenarios](https://datadoghq.atlassian.net/browse/APMSP-2291)

    ## Implementation details

    ### Benchmark design changes (what reduced flakiness)

    1. Run every benchmark 10 times (`launchCount`).
    - This is important to mitigate inter-run variance, which can be caused
    by things such as varying memory placement across runs.
    2. Manually set warmup iterations to 10 (`warmupCount`).
    - On average, the number of automatically determined warmup iterations
    is ~15 for the current benchmarking setup.
        - I got this down to 10 because:
            - I needed to make benchmarks faster.
    - On average, every iteration runs the benchmark method for hundreds of
    thousands of times, so I hypothesized that 5 less warmup iterations
    would not affect results significantly.
    - We were already doing 10 runs. I hypothesized that stability would
    come from different process samples, not from a deeper warmup within a
    single process.
    3. Manually set actual iterations to 10 (`iterationCount`).
    - On average, the number of automatically determined warmup iterations
    is ~35 for the current benchmarking setup. Some
    (`CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces`,
    `StringConcatAspectBenchmark`, `StringConcatBenchmark`) go up to 120.
    - I got this down to 10 because of the same reasons described on change
    2.
    4. Kept a "base" `iterationTime` of 200 ms, but set `iterationTime` on
    flaky benchmarks to 500 ms.
    - Increasing the `iterationTime` for flaky benchmarks helps immensely.
    We're able to capture much more variation into our samples once we fit
    more operations into the iteration. I tried many configurations, this
    turned out to be the minimum we need.

    ### Benchmark execution changes (what allowed me to implement the
    benchmark design changes)

    1. Moved code from [benchmarking-platform @
    dd-trace-dotnet/micro](https://github.com/DataDog/benchmarking-platform/tree/dd-trace-dotnet/micro)
    to here.
    - This is just easier to understand, and doesn't require changing two
    repos to update benchmarks.
    - ⚠️ We just have to move the `ami.yaml` to build the AMIs with which we
    run microbenchmarks. But there's some preliminary work to avoid
    dd-octo-sts failures on this repo. Captured on
    [APMSP-2736](https://datadoghq.atlassian.net/browse/APMSP-2736).
    2. Used `bp-runner` to run the benchmarks, since it allows us to run
    microbenchmarks in parallel on isolated CPU sets.
    - This was crucial to allow us to run 10 launches. The current
    benchmarking job takes 50 minutes (20 min build + 30 min benchmarks). By
    running things 10 times sequentially it would take us 320 minutes.
    3. Split benchmarks into two batches: flaky ones and non-flaky ones. I
    moved the non-flaky ones to CPUs in NUMA node 1 because Windows often
    runs system processes on the first NUMA node, which can affect results.

    ### Benchmark class/Build.cs changes

    1. `Build.cs`:
    - Added a `BuildBenchmarks` target to build all benchmark executables in
    one go (needed for `bp-runner` parallel execution).
    2. **`AppSecWafBenchmark.cs` / `AppSecBenchmarkUtils.cs`**: Fixed path
    resolution to make .NET Framework 4.7.2 benchmarks run in parallel.
    - Changed `Directory.GetCurrentDirectory()` and
    `Environment.CurrentDirectory` to `AppContext.BaseDirectory`.
    - BenchmarkDotNet spawns a separate process for each benchmark run, and
    on .NET Framework the working directory doesn't match the executable
    directory. This caused WAF initialization to fail with "No valid rules
    found" because the rules file and native libraries couldn't be located.
    - Added `[IterationTime(500)]` attribute to benchmarks identified as
    flaky (e.g., `AspNetCoreBenchmark`).

    ## Test coverage

    - [x] CI job succeeds:
    https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394
    - [x] Results are uploaded to S3:
    https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394#L3875
    - [x] Results are uploaded to the BP UI:
    https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1524793394#L4200
    and
    https://benchmarking.us1.prod.dog/benchmarks?projectId=9&ciJobDateStart=1771416292448&ciJobDateEnd=1774008292448&gitBranch=augusto%2Ffix-microbenchmarks&benchmarkGroupPipelineId=103686524&benchmarkGroupSha=ce6c9b12

    ## Other details
    <!-- Fixes #{issue} -->

    ### How did we measure flakiness?

    1. Run the `run-benchmarks` job for the same commit 10 times.
    2. Cross-validate results.
    - For every benchmark and metric (`execution_time`, `throughtput`, and
    `allocated_mem`✨), we compute the **_false positive rate_** and the
    **_max confidence interval difference_**.
    - The **_false positive rate_** is how often the benchmark system
    incorrectly flags a "regression" when comparing identical code.
    - The **_max confidence interval difference_** is the largest spurious
    difference observed across all 45 comparisons of `run-benchmarks`
    results when comparing identical code.

    ### How much did flakiness drop?

    - The false positive rate  dropped from 5.7% to 3.8%.
    - The maximum CI difference dropped from 56% to 16.66%.

    See the [per-benchmark/metric
    breakdown](https://docs.google.com/spreadsheets/d/103hjAaXOBSbwSHfD-nhBoM23j8dLdeenRMGAS2TeBGU/edit?usp=sharing)

    This graph shows intuitively the distribution of FP rates and max CI
    differences for all comparisons. **The more weight the distribution has
    closer to 0, the better.** Memory allocation results are already pretty
    stable, so it's hard to see a clear win there.

    <img width="1213" height="601" alt="image"
    src="https://github.com/user-attachments/assets/b95e8d0d-1092-48e0-8ff7-ceda61174a63"
    />

    ### How were flaky benchmarks determined (necessary to choose a higher
    `iterationTime` for them)?

    1. Started with existing ignored list @NachoEchevarria and I created.
    2. Ran stability experiments and refined based on data (again,
    [per-benchmark/metric
    breakdown](https://docs.google.com/spreadsheets/d/103hjAaXOBSbwSHfD-nhBoM23j8dLdeenRMGAS2TeBGU/edit?usp=sharing))
    - AppSec benchmarks (`AppSecWafBenchmark`, `AppSecBodyBenchmark`): Moved
    to stable batch. With the new design, most FP rates are under 5%, only
    one outlier at 20% for the `netcoreapp3.1` runtime.
    - `AspNetCoreBenchmark`: Consistently over 20% FP even with optimized
    design.
    - `AgentWriterBenchmark`: Added to flaky batch. Most methods are stable,
    but `WriteAndFlushEnrichedTraces netcoreapp3.1` is VERY problematic.

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

    [APMSP-2736]:
    https://datadoghq.atlassian.net/browse/APMSP-2736?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

commit 275c3a4b037764154277a7cea2a275ab788751d7
Author: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
Date:   Tue Apr 7 13:47:42 2026 -0400

    [Version Bump] 3.42.0 (#8419)

    The following files were found to be modified (as expected)

    - [x] docs/CHANGELOG.md
    - [x] .azure-pipelines/ultimate-pipeline.yml
    - [x]
    profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/CMakeLists.txt
    - [x]
    profiler/src/ProfilerEngine/Datadog.Profiler.Native.Windows/Resource.rc
    - [x]
    profiler/src/ProfilerEngine/Datadog.Profiler.Native/dd_profiler_version.h
    - [x]
    profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/CMakeLists.txt
    - [x] profiler/src/ProfilerEngine/ProductVersion.props
    - [x] shared/src/Datadog.Trace.ClrProfiler.Native/CMakeLists.txt
    - [x] shared/src/Datadog.Trace.ClrProfiler.Native/Resource.rc
    - [x] shared/src/msi-installer/WindowsInstaller.wixproj
    - [x] shared/src/native-src/version.h
    - [x] tracer/build/artifacts/dd-dotnet.sh
    - [x] tracer/build/_build/Build.cs
    - [x]
    tracer/samples/AutomaticTraceIdInjection/MicrosoftExtensionsExample/MicrosoftExtensionsExample.csproj
    - [x]
    tracer/samples/AutomaticTraceIdInjection/Log4NetExample/Log4NetExample.csproj
    - [x]
    tracer/samples/AutomaticTraceIdInjection/NLog40Example/NLog40Example.csproj
    - [x]
    tracer/samples/AutomaticTraceIdInjection/NLog45Example/NLog45Example.csproj
    - [x]
    tracer/samples/AutomaticTraceIdInjection/NLog46Example/NLog46Example.csproj
    - [x]
    tracer/samples/AutomaticTraceIdInjection/SerilogExample/SerilogExample.csproj
    - [x] tracer/samples/ConsoleApp/Alpine3.10.dockerfile
    - [x] tracer/samples/ConsoleApp/Alpine3.9.dockerfile
    - [x] tracer/samples/ConsoleApp/Debian.dockerfile
    - [x] tracer/samples/OpenTelemetry/Debian.dockerfile
    - [x] tracer/samples/WindowsContainer/Dockerfile
    - [x] tracer/src/Datadog.Trace.ClrProfiler.Managed.Loader/Startup.cs
    - [x] tracer/src/Datadog.Tracer.Native/CMakeLists.txt
    - [x] tracer/src/Datadog.Tracer.Native/dd_profiler_constants.h
    - [x] tracer/src/Datadog.Tracer.Native/Resource.rc
    - [x] tracer/src/Directory.Build.props
    - [x] tracer/src/Datadog.Trace/TracerConstants.cs

    @DataDog/apm-dotnet

    Co-authored-by: link04 <[email protected]>

commit 84efe2bfa576e95bbb385bbeb5832ee5ccc51648
Author: Raphaël Vandon <[email protected]>
Date:   Fri Apr 3 07:00:29 2026 +0100

    maybe fix macos smoketests (#8413)

    ## Summary of changes

    ## Reason for change

    ## Implementation details

    ## Test coverage

    ## Other details
    <!-- Fixes #{issue} -->

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit 09325736fb3fb2c626d25e94b8e970a7fa02459f
Author: Zach Montoya <[email protected]>
Date:   Thu Apr 2 15:54:55 2026 -0700

    [Tracing] Update mapping from Datadog to OTLP spans (#8385)

    ## Summary of changes
    Add core Datadog semantics as span attributes. This ensures that the
    core semantics are identified when the Datadog Agent / backend receives
    OTLP spans and translates them into Datadog spans that power the
    backend.

    ## Reason for change
    We should keep the same Datadog experience regardless of the tracing
    protocol used. These changes facilitate that.

    ## Implementation details
    Updates `Datadog.Trace.OpenTelemetry.OtlpMapper.EmitAttributesFromSpan`
    to add several span attributes sourced from `Datadog.Trace.Span`
    properties:
    - `Span.ServiceName` => `OtlpSpan.Attributes["service.name"]`
    - `Span.OperationName` => `OtlpSpan.Attributes["operation.name"]`
    - `Span.ResourceName` => `OtlpSpan.Attributes["resource.name"]`
    - `Span.Type` => `OtlpSpan.Attributes["span.type"]`

    ## Test coverage
    - Unit tests: Added unit tests for
    `Datadog.Trace.OpenTelemetry.OtlpMapper` which handles the mapping from
    Datadog-specific concepts into OTLP attributes
    - Integration tests: Updated the snapshot for `OpenTelemetrySdkTests`

    ## Other details
    N/A

commit 32c92b2a10d2299f612d41a89e34c16f18ae5273
Author: Lucas Pimentel <[email protected]>
Date:   Thu Apr 2 15:57:58 2026 -0400

    [Tests] Use `EnvironmentRestorer` to prevent env var leaks (#8388)

    ## Summary of changes

    Add `[EnvironmentRestorer]` attribute to test classes/methods that call
    `Environment.SetEnvironmentVariable` without proper cleanup, preventing
    environment variable leaks between tests.

    ## Reason for change

    Several test files were setting environment variables without restoring
    them, which could cause flaky or order-dependent test failures due to
    leaked state.

    ## Implementation details

    - **`EnvironmentRestorerAttribute`**: Allow usage on methods
    (`AttributeTargets.Method`) in addition to classes
    - **`TelemetrySettingsAgentlessSettingsTests`**: Replace manual
    `IDisposable` save/restore with class-level `[EnvironmentRestorer]`
    - **`AzurePerformanceCountersListenerTests`**: Add class-level
    `[EnvironmentRestorer("WEBSITE_COUNTERS_CLR")]` (previously had no
    restore at all)
    - **`LegacyCommandLineArgumentsTests.SetCi`**: Add method-level
    `[EnvironmentRestorer("TF_BUILD")]`, remove manual try/finally
    - **`ConfigureCiCommandTests.ConfigureCi`**: Add method-level
    `[EnvironmentRestorer("GITHUB_ENV")]`, remove manual save/restore
    - `AutodetectCi` keeps its manual try/finally because it clears all
    environment variables, which the attribute can't handle

    ## Test coverage

    Existing tests cover the affected functionality. No new tests needed —
    this is a cleanup of test infrastructure.

    ## Other details

    > *"I tried to leak an environment variable once, but it was restored
    before anyone noticed."* — Claude 🤖

    ---------

    Co-authored-by: Andrew Lock <[email protected]>

commit 5686c1c6af5f1e8fa3724aa82ddbd7f29ba90b43
Author: Andrew Lock <[email protected]>
Date:   Thu Apr 2 20:12:30 2026 +0100

    Add potential fix for duck typing derived types (#8410)

    ## Summary of changes

    Potentially fixes a `FieldAccessException` in some cases

    ## Reason for change

    We have seen stack traces like the following in production:

    ```bash
    Error : Error creating or populating span.
    System.FieldAccessException
       at REDACTED
       at Datadog.Trace.ClrProfiler.ScopeFactory.CreateInactiveOutboundHttpSpan(Tracer tracer, String httpMethod, Uri requestUri, IntegrationId integrationId, HttpTags& tags, TraceId traceId, UInt64 spanId, Nullable`1 startTime, Boolean addToTraceContext)
    ```

    After a bunch of 🤖 noodling and testing (and based on @zacharycmontoya
    idea), proved that we can repro this stack (sort of, assuming inlining)
    if we _derive_ from `System.Uri`. The error happens because we apply
    `IgnoresAccessChecksToAttribute` to the target type (i.e. the derived
    type) during ducktyping, but it _needs_ to be applied to the
    `targetField.DeclaringType`.

    ## Implementation details

    Updated the `UseDirectAccessTo` calls to point to the type that "owns"
    the field, instead of the current instance.

    ## Test coverage

    Added a test confirming the error, and then showing it passed

    ## Other details

    https://app.datadoghq.com/cases/APMLP-35

commit ef71061988a026654c7e784732980f8d468499ad
Author: Andrew Lock <[email protected]>
Date:   Thu Apr 2 19:13:17 2026 +0100

    Revert YARP exclusion (#8407)

    ## Summary of changes

    Reverts including YARP spans in the ignore handler

    ## Reason for change

    We were a bit hasty including this in #8370

    ## Implementation details

    Remove it again

    ## Test coverage

    Explicitly disabling OTel in the test instead for now. There's nothing
    _wrong_ with the span, just inconsistency between versions. We could
    consider having different snapshots, but I'll do that as part of a
    follow up, as its important to get this in fast

commit bd1574507e3588ee7743ee4befcccf0357956f3d
Author: Steven Bouwkamp <[email protected]>
Date:   Thu Apr 2 11:07:30 2026 -0400

    Skip some errors from being sent to telemetry (#8003)

    ## Summary of changes

    Skips a few errors that are `Ignored` in Error Tracking and shouldn't
    really be sent to telemetry as we can't act on them.

    ## Reason for change

    Nothing we can really do about these.

    ## Implementation details

    Marked as `ErrorSkipTelemetry`

    ## Test coverage

    N/A

    ## Other details
    <!-- Fixes #{issue} -->

    I guess one thing we do lose here is whether we _want_ to see how often
    these happen?

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit 291ef8e6f8c27b2bdc422dfd218888a4e07a2c7c
Author: Lucas Pimentel <[email protected]>
Date:   Thu Apr 2 11:01:47 2026 -0400

    [CI Tools] Add prerequisite validation to AzureDevOps helper scripts (#8409)

    ## Summary of changes

    Add a `Test-Prerequisites` function to `AzureDevOpsHelpers.psm1` that
    validates all required CLI tools are installed, authenticated, and
    properly configured before running Azure DevOps build analysis or retry
    scripts.

    ## Reason for change

    When users run `Get-AzureDevOpsBuildAnalysis.ps1` or
    `Retry-AzureDevOpsFailedStages.ps1` directly (not via the Claude skill),
    missing or misconfigured prerequisites produce unhelpful error messages.
    The Claude skill had guidance for these scenarios, but the scripts
    themselves only checked if `az` and `gh` were installed.

    ## Implementation details

    New `Test-Prerequisites` function checks (in order):
    1. **`az` CLI installed** — with install links for Windows/macOS
    2. **`azure-devops` extension** — with `az extension add` command
    3. **`az` authenticated** — with `az login` guidance (including MFA
    hint)
    4. **Subscription logging** — logs current subscription at `Verbose`
    level for troubleshooting
    5. **`gh` CLI installed** — only when needed for PR-based resolution
    6. **`gh` authenticated** — scoped to `github.com` only, so GHES auth
    issues don't block

    `Resolve-BuildId` (the shared entry point for both scripts) now
    delegates to `Test-Prerequisites` instead of inline `Get-Command`
    checks.

    Additionally, `Invoke-AzDevOpsApi` now includes a troubleshooting tip in
    its error message suggesting the user check and switch Azure
    subscriptions if needed.

    ## Test coverage

    Tested manually:
    - [x] `az` not installed
    - [x] `az` installed without `azure-devops` extension
    - [x] `az` installed but not authenticated
    - [x] `gh` not installed
    - [x] `gh` installed but not authenticated

    ## Other details

    The subscription is not validated upfront because `az devops invoke
    --org <url> --detect false` targets the org URL directly, so the
    subscription doesn't strictly control API routing. However, the wrong
    subscription can affect token permissions, so `Invoke-AzDevOpsApi` now
    suggests checking subscriptions in its error message.

    > *"I validate your prerequisites so you don't have to validate your
    life choices."* — Claude 🤖

    ---------

    Co-authored-by: Copilot <[email protected]>

commit 94d461d7e1a32e617ef723764c6c5275c19fcef3
Author: Raphaël Vandon <[email protected]>
Date:   Thu Apr 2 16:00:53 2026 +0100

    scrub process tags from debugger snapshots (#8406)

    ## Summary of changes

    follow up on https://github.com/DataDog/dd-trace-dotnet/pull/8296
    debugger itests didn't run in the PR
    so we missed the fact that we need to apply the same scrubbing we did on
    other tests for those too

    ## Reason for change

    ## Implementation details

    ## Test coverage

    ## Other details
    <!-- Fixes #{issue} -->

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit a57df1c432ebf06bec5c71b9aa2a3581b4cd9090
Author: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com>
Date:   Thu Apr 2 11:45:53 2026 +0000

    chore(ci) update one-pipeline (#8403)

    This pull request updates one-pipeline to a newer version.

    Recent changes:
    Add CA certificates to SSI images using BusyBox, to prevent TLS failures
    (See https://github.com/DataDog/libdatadog-build/pull/194)

    Some of these changes may have already applied depending on your
    previous version of one-pipeline. See the libdatadog-build repository
    for all changes

    Co-authored-by: gh-worker-campaigns-3e9aa4[bot] <244854796+gh-worker-campaigns-3e9aa4[bot]@users.noreply.github.com>

commit b68393db419d884599a96f4a989da06e4638d1a4
Author: Raphaël Vandon <[email protected]>
Date:   Thu Apr 2 10:33:42 2026 +0100

    switch process tags on by default (#8296)

    ## Summary of changes

    all features have been implemented, we can now GA this to all customers

    ## Reason for change

    ## Implementation details

    depends on
    https://github.com/DataDog/dd-trace-dotnet/pull/8061
    https://github.com/DataDog/dd-trace-dotnet/pull/8163
    https://github.com/DataDog/dd-trace-dotnet/pull/8295
    theoretically, https://github.com/DataDog/dd-trace-dotnet/pull/8282 as
    well

    ## Test coverage

    ## Other details
    <!-- Fixes #{issue} -->

    I took the liberty to refactor the asserts in AgentWriterTest to use
    actual asserts instead of asserting through the mock's `Verify`, so that
    we get more actionable errors when the test fails

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit 57de0e2dfa7ecc06ca1608bd5fd2041f9fa8d76a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Apr 2 08:52:55 2026 +0000

    Bump the gh-actions-packages group across 2 directories with 5 updates (#8404)

    Bumps the gh-actions-packages group with 4 updates in the / directory:
    [actions/setup-dotnet](https://github.com/actions/setup-dotnet),
    [DataDog/dd-octo-sts-action](https://github.com/datadog/dd-octo-sts-action),
    [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action)
    and [github/codeql-action](https://github.com/github/codeql-action).
    Bumps the gh-actions-packages group with 1 update in the
    /.github/actions/publish-debug-symbols directory:
    [actions/setup-go](https://github.com/actions/setup-go).

    Updates `actions/setup-dotnet` from 5.1.0 to 5.2.0
    <details>
    <summary>Release notes</summary>
    <p><em>Sourced from <a
    href="https://github.com/actions/setup-dotnet/releases">actions/setup-dotnet's
    releases</a>.</em></p>
    <blockquote>
    <h2>v5.2.0</h2>
    <h2>What's changed</h2>
    <h3>Enhancements</h3>
    <ul>
    <li>Add support for workloads input by <a
    href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a> in
    <a
    href="https://redirect.github.com/actions/setup-dotnet/pull/693">actions/setup-dotnet#693</a></li>
    <li>Add support for optional architecture input for cross-architecture
    .NET installs by <a
    href="https://github.com/priya-kinthali"><code>@​priya-kinthali</code></a>
    in <a
    href="https://redirect.github.com/actions/setup-dotnet/pull/700">actions/setup-dotnet#700</a></li>
    </ul>
    <h3>Dependency Updates</h3>
    <ul>
    <li>Upgrade fast-xml-parser from 4.4.1 to 5.3.6 by <a
    href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
    href="https://redirect.github.com/actions/setup-dotnet/pull/671">actions/setup-dotnet#671</a></li>
    <li>Upgrade minimatch from 3.1.2 to 3.1.5 by <a
    href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
    href="https://redirect.github.com/actions/setup-dotnet/pull/705">actions/setup-dotnet#705</a></li>
    </ul>
    <p><strong>Full Changelog</strong>: <a
    href="https://github.com/actions/setup-dotnet/compare/v5...v5.2.0">https://github.com/actions/setup-dotnet/compare/v5...v5.2.0</a></p>
    </blockquote>
    </details>
    <details>
    <summary>Commits</summary>
    <ul>
    <li><a
    href="https://github.com/actions/setup-dotnet/commit/c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7"><code>c2fa09f</code></a>
    Bump minimatch from 3.1.2 to 3.1.5 (<a
    href="https://redirect.github.com/actions/setup-dotnet/issues/705">#705</a>)</li>
    <li><a
    href="https://github.com/actions/setup-dotnet/commit/02574b18e2dc57a218ee4e11ba1e1603c67236e8"><code>02574b1</code></a>
    Add support for optional architecture input for cross-architecture .NET
    insta...</li>
    <li><a
    href="https://github.com/actions/setup-dotnet/commit/16c7b3c2fa55a0e394467d22512b84fda46adf63"><code>16c7b3c</code></a>
    Bump fast-xml-parser from 4.4.1 to 5.3.6 (<a
    href="https://redirect.github.com/actions/setup-dotnet/issues/671">#671</a>)</li>
    <li><a
    href="https://github.com/actions/setup-dotnet/commit/131b410979e0b49e2162c0718030257b22d6dc2c"><code>131b410</code></a>
    Add support for workloads input (<a
    href="https://redirect.github.com/actions/setup-dotnet/issues/693">#693</a>)</li>
    <li>See full diff in <a
    href="https://github.com/actions/setup-dotnet/compare/v5.1.0...c2fa09f4bde5ebb9d1777cf28262a3eb3db3ced7">compare
    view</a></li>
    </ul>
    </details>
    <br />

    Updates `DataDog/dd-octo-sts-action` from 1.0.3 to 1.0.4
    <details>
    <summary>Commits</summary>
    <ul>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a"><code>96a2546</code></a>
    Fix typo in Readme (<a
    href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/18">#18</a>)</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/9691c26e1de0f1f26e1e8708c5c34b4f64e43f5f"><code>9691c26</code></a>
    Merge pull request <a
    href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/14">#14</a>
    from DataDog/improve/parse-jwt-claims</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/b98b59d08d3575cbda7001bddfe86633787536e8"><code>b98b59d</code></a>
    Merge pull request <a
    href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/13">#13</a>
    from DataDog/improve/fetch-error-url-logging</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/e7953d4e870e933635e6afa9172b3957b568c417"><code>e7953d4</code></a>
    Merge pull request <a
    href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/15">#15</a>
    from DataDog/improve/ci-workflow-hardening</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/e47344e9570a80d3a7d333a339ace4a5e88b7646"><code>e47344e</code></a>
    Merge pull request <a
    href="https://redirect.github.com/datadog/dd-octo-sts-action/issues/16">#16</a>
    from DataDog/improve/bump-node24</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/5a7a632cb3be2334cd1515df9c74eb3103942b50"><code>5a7a632</code></a>
    Bump Node.js runtime from node20 to node24</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/260fcf964ad38660b2abc359216586af9d31a05d"><code>260fcf9</code></a>
    Add parseJwtClaims() function with tests, replace fragile inline
    parsing</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/371c4d81ebd5ed74dfcc7bb2ab234d9f1e30fe65"><code>371c4d8</code></a>
    Harden CI workflows with least-privilege permissions and credential
    controls</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/1fc658893bed0edd73a7e284f6266e3fc4bdc93e"><code>1fc6588</code></a>
    Include URL in fetchWithRetry error messages</li>
    <li><a
    href="https://github.com/DataDog/dd-octo-sts-action/commit/0b31f95da950c7562ef40f6447086e75515897ce"><code>0b31f95</code></a>
    Harden CI workflows with least-privilege permissions and credential
    controls</li>
    <li>Additional commits viewable in <a
    href="https://github.com/datadog/dd-octo-sts-action/compare/v1.0.3...96a25462dbcb10ebf0bfd6e2ccc917d2ab235b9a">compare
    view</a></li>
    </ul>
    </details>
    <br />

    Updates `slackapi/slack-github-action` from 2.1.1 to 3.0.1
    <details>
    <summary>Release notes</summary>
    <p><em>Sourced from <a
    href="https://github.com/slackapi/slack-github-action/releases">slackapi/slack-github-action's
    releases</a>.</em></p>
    <blockquote>
    <h2>Slack GitHub Action v3.0.1</h2>
    <h2>What's Changed</h2>
    <p>Alongside the breaking changes of <a
    href="https://github.com/slackapi/slack-github-action/releases/tag/v3.0.0"><code>@v3.0.0</code></a>
    and a <a
    href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/">new
    technique</a> to run Slack CLI commands, we tried the wrong name to
    publish to the GitHub Marketplace 🐙 This action is now noted as <a
    href="https://github.com/marketplace/actions/the-slack-github-action"><strong>The
    Slack GitHub Action</strong></a> in listings 🎶 ✨</p>
    <h3>:art: Maintenance</h3>
    <ul>
    <li>chore: use a unique title for marketplace in <a
    href="https://redirect.github.com/slackapi/slack-github-action/pull/576">slackapi/slack-github-action#576</a>
    - Thanks <a
    href="https://github.com/zimeg"><code>@​zimeg</code></a>!</li>
    <li>chore(release): tag version 3.0.1 in <a
    href="https://redirect.github.com/slackapi/slack-github-action/pull/577">slackapi/slack-github-action#577</a>
    - Thanks <a
    href="https://github.com/zimeg"><code>@​zimeg</code></a>!</li>
    </ul>
    <p><strong>Full Changelog</strong>: <a
    href="https://github.com/slackapi/slack-github-action/compare/v3.0.0...v3.0.1">https://github.com/slackapi/slack-github-action/compare/v3.0.0...v3.0.1</a></p>
    <h2>Slack GitHub Action v3.0.0</h2>
    <blockquote>
    <p>The <code>@v3.0.0</code> release had a hiccup on publish and we
    recommend using <a
    href="https://github.com/slackapi/slack-github-action/releases/tag/v3.0.1"><strong><code>@​v3.0.1</code></strong></a>
    or a more recent version when updating! Oops!</p>
    </blockquote>
    <p>🎽 <strong>Running Slack CLI commands and the active Node runtime,
    both included in this release</strong> 👟 ✨</p>
    <h3>⚠️ Breaking change: Node.js 24 the runtime</h3>
    <p>This major version updates the GitHub Actions required runtime to <a
    href="https://nodejs.org/en/about/previous-releases"><strong>Node.js
    24</strong>.</a> Most <a
    href="https://github.com/actions/runner-images?tab=readme-ov-file#software-and-image-support">GitHub-hosted
    runners</a> already include this, but self-hosted runners may need to be
    updated ahead of <a
    href="https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/">planned
    deprecations of Node 20 on GitHub Actions runners</a>.</p>
    <h3>📺 Enhancement: Run Slack CLI commands</h3>
    <p>This release introduces a new technique for running <a
    href="https://docs.slack.dev/tools/slack-cli">Slack CLI</a> commands
    directly in GitHub Actions workflows. Use this to install the latest
    version (or a specific one) of the CLI and execute commands like
    <code>deploy</code> for merges to main, <code>manifest validate</code>
    with tests, and other <a
    href="https://docs.slack.dev/tools/slack-cli/reference/commands/slack">commands</a>.</p>
    <p>Gather a token using the following CLI command to store with repo
    secrets, then get started with an example below:</p>
    <pre><code>$ slack auth token
    </code></pre>
    <h3>🧪 Validate an app manifest on pull requests</h3>
    <p>Check that your app manifest is valid before merging changes:</p>
    <p>🔗 <a
    href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/validate-a-manifest">https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/validate-a-manifest</a></p>
    <pre lang="yaml"><code>- name: Validate the manifest
      uses: slackapi/slack-github-action/[email protected]
      with:
    command: &quot;manifest validate --app ${{ vars.SLACK_APP_ID }}&quot;
        token: ${{ secrets.SLACK_SERVICE_TOKEN }}
    </code></pre>
    <h3>🚀 Deploy your app on push to main</h3>
    <p>Automate deployments whenever changes land on your main branch:</p>
    <p>🔗 <a
    href="https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/deploy-an-app">https://docs.slack.dev/tools/slack-github-action/sending-techniques/running-slack-cli-commands/deploy-an-app</a></p>
    <!-- raw HTML omitted -->
    </blockquote>
    <p>... (truncated)</p>
    </details>
    <details>
    <summary>Commits</summary>
    <ul>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/af78098f536edbc4de71162a307590698245be95"><code>af78098</code></a>
    Release</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/add1a00063f351e4c0e55c3703da81637f03a8be"><code>add1a00</code></a>
    chore(release): tag version 3.0.1 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/577">#577</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/2bc9e7a4cd10f4d06ef49b8fa8a11efdc7fb891b"><code>2bc9e7a</code></a>
    chore: use a unique title for marketplace (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/576">#576</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/c5d43dad17bba7ebd47486137b9ab6936fd6bbf4"><code>c5d43da</code></a>
    chore(release): tag version 3.0.0 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/575">#575</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/963b9796dcc3184602a0aefe2f052d034027bfaf"><code>963b979</code></a>
    build(deps): bump <code>@​slack/web-api</code> from 7.14.1 to 7.15.0 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/574">#574</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/90b7328a4cea35bd9dc6fc64d7f70e772d6d5876"><code>90b7328</code></a>
    build(deps): bump <code>@​slack/logger</code> from 4.0.0 to 4.0.1 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/573">#573</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/e45cb891a61f925570820f137980df2028625fec"><code>e45cb89</code></a>
    feat: support slack cli commands with composite action inputs (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/560">#560</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/0aed2c2a70fe17c67bfd489b5dc3d9b410f69f79"><code>0aed2c2</code></a>
    build(deps): bump https-proxy-agent from 7.0.6 to 8.0.0 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/572">#572</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/4795f96c2818074349810cac0abc3bf5437bdc2c"><code>4795f96</code></a>
    build(deps-dev): bump sinon from 21.0.1 to 21.0.2 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/571">#571</a>)</li>
    <li><a
    href="https://github.com/slackapi/slack-github-action/commit/bd9e2ce619554772120b8cfcbbc7fe4bd2d42a2f"><code>bd9e2ce</code></a>
    build(deps): bump actions/setup-node from 6.2.0 to 6.3.0 (<a
    href="https://redirect.github.com/slackapi/slack-github-action/issues/569">#569</a>)</li>
    <li>Additional commits viewable in <a
    href="https://github.com/slackapi/slack-github-action/compare/91efab103c0de0a537f72a35f6b8cda0ee76bf0a...af78098f536edbc4de71162a307590698245be95">compare
    view</a></li>
    </ul>
    </details>
    <br />

    Updates `github/codeql-action` from 4.34.1 to 4.35.1
    <details>
    <summary>Release notes</summary>
    <p><em>Sourced from <a
    href="https://github.com/github/codeql-action/releases">github/codeql-action's
    releases</a>.</em></p>
    <blockquote>
    <h2>v4.35.1</h2>
    <ul>
    <li>Fix incorrect minimum required Git version for <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a>: it should have been 2.36.0, not 2.11.0. <a
    href="https://redirect.github.com/github/codeql-action/pull/3781">#3781</a></li>
    </ul>
    <h2>v4.35.0</h2>
    <ul>
    <li>Reduced the minimum Git version required for <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a> from 2.38.0 to 2.11.0. <a
    href="https://redirect.github.com/github/codeql-action/pull/3767">#3767</a></li>
    <li>Update default CodeQL bundle version to <a
    href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.1">2.25.1</a>.
    <a
    href="https://redirect.github.com/github/codeql-action/pull/3773">#3773</a></li>
    </ul>
    </blockquote>
    </details>
    <details>
    <summary>Changelog</summary>
    <p><em>Sourced from <a
    href="https://github.com/github/codeql-action/blob/main/CHANGELOG.md">github/codeql-action's
    changelog</a>.</em></p>
    <blockquote>
    <h1>CodeQL Action Changelog</h1>
    <p>See the <a
    href="https://github.com/github/codeql-action/releases">releases
    page</a> for the relevant changes to the CodeQL CLI and language
    packs.</p>
    <h2>[UNRELEASED]</h2>
    <ul>
    <li>The Git version 2.36.0 requirement for improved incremental analysis
    now only applies to repositories that contain submodules. <a
    href="https://redirect.github.com/github/codeql-action/pull/3789">#3789</a></li>
    <li>Python analysis on GHES no longer extracts the standard library,
    relying instead on models of the standard library. This should result in
    significantly faster extraction and analysis times, while the effect on
    alerts should be minimal. <a
    href="https://redirect.github.com/github/codeql-action/pull/3794">#3794</a></li>
    </ul>
    <h2>4.35.1 - 27 Mar 2026</h2>
    <ul>
    <li>Fix incorrect minimum required Git version for <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a>: it should have been 2.36.0, not 2.11.0. <a
    href="https://redirect.github.com/github/codeql-action/pull/3781">#3781</a></li>
    </ul>
    <h2>4.35.0 - 27 Mar 2026</h2>
    <ul>
    <li>Reduced the minimum Git version required for <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a> from 2.38.0 to 2.11.0. <a
    href="https://redirect.github.com/github/codeql-action/pull/3767">#3767</a></li>
    <li>Update default CodeQL bundle version to <a
    href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.1">2.25.1</a>.
    <a
    href="https://redirect.github.com/github/codeql-action/pull/3773">#3773</a></li>
    </ul>
    <h2>4.34.1 - 20 Mar 2026</h2>
    <ul>
    <li>Downgrade default CodeQL bundle version to <a
    href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.24.3">2.24.3</a>
    due to issues with a small percentage of Actions and JavaScript
    analyses. <a
    href="https://redirect.github.com/github/codeql-action/pull/3762">#3762</a></li>
    </ul>
    <h2>4.34.0 - 20 Mar 2026</h2>
    <ul>
    <li>Added an experimental change which disables TRAP caching when <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a> is enabled, since improved incremental analysis
    supersedes TRAP caching. This will improve performance and reduce
    Actions cache usage. We expect to roll this change out to everyone in
    March. <a
    href="https://redirect.github.com/github/codeql-action/pull/3569">#3569</a></li>
    <li>We are rolling out improved incremental analysis to C/C++ analyses
    that use build mode <code>none</code>. We expect this rollout to be
    complete by the end of April 2026. <a
    href="https://redirect.github.com/github/codeql-action/pull/3584">#3584</a></li>
    <li>Update default CodeQL bundle version to <a
    href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.25.0">2.25.0</a>.
    <a
    href="https://redirect.github.com/github/codeql-action/pull/3585">#3585</a></li>
    </ul>
    <h2>4.33.0 - 16 Mar 2026</h2>
    <ul>
    <li>
    <p>Upcoming change: Starting April 2026, the CodeQL Action will skip
    collecting file coverage information on pull requests to improve
    analysis performance. File coverage information will still be computed
    on non-PR analyses. Pull request analyses will log a warning about this
    upcoming change. <a
    href="https://redirect.github.com/github/codeql-action/pull/3562">#3562</a></p>
    <p>To opt out of this change:</p>
    <ul>
    <li><strong>Repositories owned by an organization:</strong> Create a
    custom repository property with the name
    <code>github-codeql-file-coverage-on-prs</code> and the type
    &quot;True/false&quot;, then set this property to <code>true</code> in
    the repository's settings. For more information, see <a
    href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">Managing
    custom properties for repositories in your organization</a>.
    Alternatively, if you are using an advanced setup workflow, you can set
    the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable
    to <code>true</code> in your workflow.</li>
    <li><strong>User-owned repositories using default setup:</strong> Switch
    to an advanced setup workflow and set the
    <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable to
    <code>true</code> in your workflow.</li>
    <li><strong>User-owned repositories using advanced setup:</strong> Set
    the <code>CODEQL_ACTION_FILE_COVERAGE_ON_PRS</code> environment variable
    to <code>true</code> in your workflow.</li>
    </ul>
    </li>
    <li>
    <p>Fixed <a
    href="https://redirect.github.com/github/codeql-action/issues/3555">a
    bug</a> which caused the CodeQL Action to fail loading repository
    properties if a &quot;Multi select&quot; repository property was
    configured for the repository. <a
    href="https://redirect.github.com/github/codeql-action/pull/3557">#3557</a></p>
    </li>
    <li>
    <p>The CodeQL Action now loads <a
    href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">custom
    repository properties</a> on GitHub Enterprise Server, enabling the
    customization of features such as
    <code>github-codeql-disable-overlay</code> that was previously only
    available on GitHub.com. <a
    href="https://redirect.github.com/github/codeql-action/pull/3559">#3559</a></p>
    </li>
    <li>
    <p>Once <a
    href="https://docs.github.com/en/code-security/how-tos/secure-at-scale/configure-organization-security/manage-usage-and-access/giving-org-access-private-registries">private
    package registries</a> can be configured with OIDC-based authentication
    for organizations, the CodeQL Action will now be able to accept such
    configurations. <a
    href="https://redirect.github.com/github/codeql-action/pull/3563">#3563</a></p>
    </li>
    <li>
    <p>Fixed the retry mechanism for database uploads. Previously this would
    fail with the error &quot;Response body object should not be disturbed
    or locked&quot;. <a
    href="https://redirect.github.com/github/codeql-action/pull/3564">#3564</a></p>
    </li>
    <li>
    <p>A warning is now emitted if the CodeQL Action detects a repository
    property whose name suggests that it relates to the CodeQL Action, but
    which is not one of the properties recognised by the current version of
    the CodeQL Action. <a
    href="https://redirect.github.com/github/codeql-action/pull/3570">#3570</a></p>
    </li>
    </ul>
    <h2>4.32.6 - 05 Mar 2026</h2>
    <ul>
    <li>Update default CodeQL bundle version to <a
    href="https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.24.3">2.24.3</a>.
    <a
    href="https://redirect.github.com/github/codeql-action/pull/3548">#3548</a></li>
    </ul>
    <h2>4.32.5 - 02 Mar 2026</h2>
    <ul>
    <li>Repositories owned by an organization can now set up the
    <code>github-codeql-disable-overlay</code> custom repository property to
    disable <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis for CodeQL</a>. First, create a custom repository
    property with the name <code>github-codeql-disable-overlay</code> and
    the type &quot;True/false&quot; in the organization's settings. Then in
    the repository's settings, set this property to <code>true</code> to
    disable improved incremental analysis. For more information, see <a
    href="https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization">Managing
    custom properties for repositories in your organization</a>. This
    feature is not yet available on GitHub Enterprise Server. <a
    href="https://redirect.github.com/github/codeql-action/pull/3507">#3507</a></li>
    <li>Added an experimental change so that when <a
    href="https://redirect.github.com/github/roadmap/issues/1158">improved
    incremental analysis</a> fails on a runner — potentially due to
    insufficient disk space — the failure is recorded in the Actions cache
    so that subsequent runs will automatically skip improved incremental
    analysis until something changes (e.g. a larger runner is provisioned or
    a new CodeQL version is released). We expect to roll this change out to
    everyone in March. <a
    href="https://redirect.github.com/github/codeql-action/pull/3487">#3487</a></li>
    </ul>
    <!-- raw HTML omitted -->
    </blockquote>
    <p>... (truncated)</p>
    </details>
    <details>
    <summary>Commits</summary>
    <ul>
    <li><a
    href="https://github.com/github/codeql-action/commit/c10b8064de6f491fea524254123dbe5e09572f13"><code>c10b806</code></a>
    Merge pull request <a
    href="https://redirect.github.com/github/codeql-action/issues/3782">#3782</a>
    from github/update-v4.35.1-d6d1743b8</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/c5ffd0683786820677d054e3505e1c5bb4b8c227"><code>c5ffd06</code></a>
    Update changelog for v4.35.1</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/d6d1743b8ec7ecd94f78ad1ce4cb3d8d2ba58001"><code>d6d1743</code></a>
    Merge pull request <a
    href="https://redirect.github.com/github/codeql-action/issues/3781">#3781</a>
    from github/henrymercer/update-git-minimum-version</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/65d2efa7333ad65f97cc54be40f4cd18630f884c"><code>65d2efa</code></a>
    Add changelog note</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/2437b20ab31021229573a66717323dd5c6ce9319"><code>2437b20</code></a>
    Update minimum git version for overlay to 2.36.0</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/ea5f71947c021286c99f61cc426a10d715fe4434"><code>ea5f719</code></a>
    Merge pull request <a
    href="https://redirect.github.com/github/codeql-action/issues/3775">#3775</a>
    from github/dependabot/npm_and_yarn/node-forge-1.4.0</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/45ceeea896ba2293e10982f871198d1950ee13d6"><code>45ceeea</code></a>
    Merge pull request <a
    href="https://redirect.github.com/github/codeql-action/issues/3777">#3777</a>
    from github/mergeback/v4.35.0-to-main-b8bb9f28</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/24448c98434f429f901d27db7ddae55eec5cc1c4"><code>24448c9</code></a>
    Rebuild</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/7c510606312e5c68ac8b27c009e5254f226f5dfa"><code>7c51060</code></a>
    Update changelog and version after v4.35.0</li>
    <li><a
    href="https://github.com/github/codeql-action/commit/b8bb9f28b8d3f992092362369c57161b755dea45"><code>b8bb9f2</code></a>
    Merge pull request <a
    href="https://redirect.github.com/github/codeql-action/issues/3776">#3776</a>
    from github/update-v4.35.0-0078ad667</li>
    <li>Additional commits viewable in <a
    href="https://github.com/github/codeql-action/compare/38697555549f1db7851b81482ff19f1fa5c4fedc...c10b8064de6f491fea524254123dbe5e09572f13">compare
    view</a></li>
    </ul>
    </details>
    <br />

    Updates `actions/setup-go` from 6.3.0 to 6.4.0
    <details>
    <summary>Release notes</summary>
    <p><em>Sourced from <a
    href="https://github.com/actions/setup-go/releases">actions/setup-go's
    releases</a>.</em></p>
    <blockquote>
    <h2>v6.4.0</h2>
    <h2>What's Changed</h2>
    <h3>Enhancement</h3>
    <ul>
    <li>Add go-download-base-url input for custom Go distributions by <a
    href="https://github.com/gdams"><code>@​gdams</code></a> in <a
    href="https://redirect.github.com/actions/setup-go/pull/721">actions/setup-go#721</a></li>
    </ul>
    <h3>Dependency update</h3>
    <ul>
    <li>Upgrade minimatch from 3.1.2 to 3.1.5 by <a
    href="https://github.com/dependabot"><code>@​dependabot</code></a> in <a
    href="https://redirect.github.com/actions/setup-go/pull/727">actions/setup-go#727</a></li>
    </ul>
    <h3>Documentation update</h3>
    <ul>
    <li>Rearrange README.md, add advanced-usage.md by <a
    href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a>
    in <a
    href="https://redirect.github.com/actions/setup-go/pull/724">actions/setup-go#724</a></li>
    <li>Fix Microsoft build of Go link by <a
    href="https://github.com/gdams"><code>@​gdams</code></a> in <a
    href="https://redirect.github.com/actions/setup-go/pull/734">actions/setup-go#734</a></li>
    </ul>
    <h2>New Contributors</h2>
    <ul>
    <li><a href="https://github.com/gdams"><code>@​gdams</code></a> made
    their first contribution in <a
    href="https://redirect.github.com/actions/setup-go/pull/721">actions/setup-go#721</a></li>
    </ul>
    <p><strong>Full Changelog</strong>: <a
    href="https://github.com/actions/setup-go/compare/v6...v6.4.0">https://github.com/actions/setup-go/compare/v6...v6.4.0</a></p>
    </blockquote>
    </details>
    <details>
    <summary>Commits</summary>
    <ul>
    <li><a
    href="https://github.com/actions/setup-go/commit/4a3601121dd01d1626a1e23e37211e3254c1c06c"><code>4a36011</code></a>
    docs: fix Microsoft build of Go link (<a
    href="https://redirect.github.com/actions/setup-go/issues/734">#734</a>)</li>
    <li><a
    href="https://github.com/actions/setup-go/commit/8f19afcc704763637be6b1718da0af52ca05785d"><code>8f19afc</code></a>
    feat: add go-download-base-url input for custom Go distributions (<a
    href="https://redirect.github.com/actions/setup-go/issues/721">#721</a>)</li>
    <li><a
    href="https://github.com/actions/setup-go/commit/27fdb267c15a8835f1ead03dfa07f89be2bb741a"><code>27fdb26</code></a>
    Bump minimatch from 3.1.2 to 3.1.5 (<a
    href="https://redirect.github.com/actions/setup-go/issues/727">#727</a>)</li>
    <li><a
    href="https://github.com/actions/setup-go/commit/def8c394e3ad351a79bc93815e4a585520fe993b"><code>def8c39</code></a>
    Rearrange README.md, add advanced-usage.md (<a
    href="https://redirect.github.com/actions/setup-go/issues/724">#724</a>)</li>
    <li>See full diff in <a
    href="https://github.com/actions/setup-go/compare/4b73464bb391d4059bd26b0524d20df3927bd417...4a3601121dd01d1626a1e23e37211e3254c1c06c">compare
    view</a></li>
    </ul>
    </details>
    <br />

    Dependabot will resolve any conflicts with this PR as long as you don't
    alter it yourself. You can also trigger a rebase manually by commenting
    `@dependabot rebase`.

    [//]: # (dependabot-automerge-start)
    [//]: # (dependabot-automerge-end)

    ---

    <details>
    <summary>Dependabot commands and options</summary>
    <br />

    You can trigger Dependabot actions by commenting on this PR:
    - `@dependabot rebase` will rebase this PR
    - `@dependabot recreate` will recreate this PR, overwriting any edits
    that have been made to it
    - `@dependabot show <dependency name> ignore conditions` will show all
    of the ignore conditions of the specified dependency
    - `@dependabot ignore <dependency name> major version` will close this
    group update PR and stop Dependabot creating any more for the specific
    dependency's major version (unless you unignore this specific
    dependency's major version or upgrade to it yourself)
    - `@dependabot ignore <dependency name> minor version` will close this
    group update PR and stop Dependabot creating any more for the specific
    dependency's minor version (unless you unignore this specific
    dependency's minor version or upgrade to it yourself)
    - `@dependabot ignore <dependency name>` will close this group update PR
    and stop Dependabot creating any more for the specific dependency
    (unless you unignore this specific dependency or upgrade to it yourself)
    - `@dependabot unignore <dependency name>` will remove all of the ignore
    conditions of the specified dependency
    - `@dependabot unignore <dependency name> <ignore condition>` will
    remove the ignore condition of the specified dependency and ignore
    conditions

    </details>

    Signed-off-by: dependabot[bot] <[email protected]>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit b40c001f54fe95c5307316b80a10a46cdcbeebbd
Author: Andrew Lock <[email protected]>
Date:   Wed Apr 1 16:13:39 2026 +0100

    Add hardcoded limits for OCI and libinjection packages (#8382)

    ## Summary of changes

    Adds .NET-specific OCI and libinjection package size limits, instead of
    relying on the "global" limits

    ## Reason for change

    https://github.com/DataDog/dd-trace-dotnet/pull/8351 bumped the global
    package size limitations for OCI and lib-injection images, but @lloeki
    flagged that this can lead to large regressions in size to slip through.
    Given that these packages are quite size-sensitive, this is sub-optimal.

    ## Implementation details

    This PR uses the work in
    https://github.com/DataDog/libdatadog-build/pull/171 and
    https://github.com/DataDog/libdatadog-build/pull/174 to set size-limits
    based on [his
    example](https://github.com/DataDog/dd-trace-rb/blob/af17de097795fc3b0053f47d1436f308e0d5f92e/.gitlab-ci.yml#L17-L18).
    It adds the package size override variables, limiting both to 40MB.

    ## Test coverage

    Right now we have the following sizes:

    lib-injection images:
    - `linux-amd64`: 38MB
    - `linux-arm64`: 35MB

    OCI images:
    - `linux-amd64`: 30MB
    - `linux-arm64`: 28MB
    - `win-amd64`: 35MB

    So a 40MB limit across the board seems reasonable to me.

    You can see these limits being applied in [the Gitlab
    run](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-dotnet/-/jobs/1545330315)

commit e72f8e8d2dc624d3ee711a78a76f113727adc0f7
Author: Andrew Lock <[email protected]>
Date:   Wed Apr 1 16:10:45 2026 +0100

    Delete the static analysis workflow (#8393)

    ## Summary of changes

    Delete the static analysis workflow

    ## Reason for change

    Nobody uses it and it's a pain to migrate

commit 954cad2f4c12a203a503401bc21774ac2ed2cc0f
Author: Steven Bouwkamp <[email protected]>
Date:   Wed Apr 1 10:47:35 2026 -0400

    Pin floating pre-release versions of SQLitePCLRaw.* dependencies in samples (#8390)

    ## Summary of changes

    Pins two different projects that were using floating versions of
    PackageReferences to `SQLitePCLRaw.bundle_e_sqlite3` and
    `SQLitePCLRaw.core`.

    These appeared to link to these versions, I bumped them to be stable

    -
    https://www.nuget.org/packages/SQLitePCLRaw.core/2.1.6-pre20230809203314
    -
    https://www.nuget.org/packages/SQLitePCLRaw.bundle_e_sqlite3/2.1.6-pre20230809203314

    ## Reason for change

    We don't want any floating dependencies

    ## Implementation details

    Searched here:
    https://github.com/search?q=repo%3ADataDog%2Fdd-trace-dotnet%20%2FVersion%3D%22%5B%5E%22%5D*%5C*%2F&type=code

    ## Test coverage

    This is the test, hopefully this works

    ## Other details
    <!-- Fixes #{issue} -->

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit 5f7ed4de8dc1152a8326d40b0ad95fb6593e6e67
Author: Steven Bouwkamp <[email protected]>
Date:   Wed Apr 1 09:48:16 2026 -0400

    Configure `GeneratePackageVersions` to support a "cooldown" on dependencies (#8371)

    ## Summary of changes

    > default cooldown period is 2 days

    Adds a configurable cooldown period to the `GeneratePackageVersions` to
    support remediation efforts for follow up after #incident-51602.

    To use supply the optional parameter `--PackageVersionCooldownDays X`
    where `X` is some number of days. The current period is at the moment is
    going to be 2 days and is the default now when the overall
    `GeneratePackageVersion` target is ran. Additionally, this is overridden
    to 0 if `--IncludePackages` is supplied (this is commonly used when
    working on a singular package locally).

    After running the tool a "cooldown" report is generated, this file will
    contain packages that we see have a newer version, but will not
    incorporate into the test, the fallback version that falls within the
    cooldown period is provided. This file content will show up in the
    output of future test package version bump PRs.

    Note that this is for _automated_ updates, so if it sees something
    already updated it will honor it.

    Here's an example output ran locally with 14 days set:

    ```
    ## Package Version Cooldown Report

    The following versions were published less than **14 days** ago and have been overridden.
    These require manual review before inclusion.

    | Package | Integration | Overridden Version | Published | Age (days) | Using Instead |
    |---------|-------------|--------------------|-----------|------------|---------------|
    | AWSSDK.Core | AwsSdk | 4.0.3.22 | 2026-03-25 | 0 | 4.0.3.21 |
    | AWSSDK.S3 | AwsS3 | 4.0.19.2 | 2026-03-25 | 0 | 4.0.19.1 |
    | StackExchange.Redis | StackExchangeRedis | 2.12.8 | 2026-03-25 | 0 | 2.12.4 |
    ```

    ## Reason for change

    In https://github.com/DataDog/dd-trace-dotnet/pull/8364 and
    #incident-51602 all automated dependency updaters to be disabled
    temporarily, to re-enable we need to supply a 2 day "cooldown" to any
    version that we update to (in other words the version of the NuGet must
    be published for at least 2 days before we can update to it).

    ## Implementation details

    I made Claude do this 🤖

    - NuGetPackageHelper now captures the Published date from
    IPackageSearchMetadata via a new VersionWithDate record (previously
    discarded)
      - NuGetVersionCache stores the new {Version, Published} format
    - PackageVersionGenerator.ApplyCooldown filters selected versions after
    LatestMajors/LatestMinors/LatestSpecific selection:
        - Versions outside the cooldown window pass through unchanged
    - Versions at or below the baseline (derived from
    supported_versions.json MaxVersionTestedInclusive) are kept even if
      within cooldown -- no downgrades
    - Versions above the baseline and within cooldown are overridden to the
    best available fallback
    - CooldownReport collects overridden versions and renders a markdown
    table saved to tracer/build/cooldown_report.md
    - The GitHub Actions workflow reads the report and appends it to the
    auto-bump PR body
    - Honeypot IntegrationGroups.cs fixes: MSTest.TestFramework now maps to
    itself, Hangfire.Core maps to Hangfire.Core (was Hangfire), OpenFeature
    mapping moved to Datadog.FeatureFlags.OpenFeature

    Passing `--IncludePackages` will override the cooldown to 0

    ## Test coverage

    I ran `GeneratePackageVersions --PackageVersionCooldownDays 14` locally
    seems good enough IMO (also ran without, with different days etc)

    ```
    [WRN] GeneratePackageVersi: 3 package version(s) were excluded due to the 14-day cooldown period
    [WRN] GeneratePackageVersi:   AWSSDK.Core 4.0.3.22 overridden (published 2026-03-25, using: 4.0.3.21)
    [WRN] GeneratePackageVersi:   AWSSDK.S3 4.0.19.2 overridden (published 2026-03-25, using: 4.0.19.1)
    [WRN] GeneratePackageVersi:   StackExchange.Redis 2.12.8 overridden (published 2026-03-25, using: 2.12.4)
    ```

    ## Other details
    <!-- Fixes #{issue} -->

    The workflow file (`auto_bump_test_package_versions.yml`) will be
    re-enabled with this PR

    <!--  ⚠️ Note:

    Where possible, please obtain 2 approvals prior to merging. Unless
    CODEOWNERS specifies otherwise, for external teams it is typically best
    to have one review from a team member, and one review from apm-dotnet.
    Trivial changes do not require 2 reviews.

    MergeQueue is NOT enabled in this repository. If you have write access
    to the repo, the PR has 1-2 approvals (see above), and all of the
    required checks have passed, you can use the Squash and Merge button to
    merge the PR. If you don't have write access, or you need help, reach
    out in the #apm-dotnet channel in Slack.
    -->

commit 07039f099ad278d0d6b397ffd7d92cea62d07cd8
Author: Andrew Lock <[email protected]>
Date:   Wed Apr 1 09:47:34 2026 +0100

    Set `DD_TRACE_OTEL_ENABLED=true` by default for all integration…
igoragoli added a commit that referenced this pull request Apr 9, 2026
…ks (#8424)

## Summary of changes

- Enable PR-level regression gates on microbenchmarks
(`check-big-regressions`).
- Enable release-level gates on macrobenchmarks (`check-slo-breaches`).
- Replace `IGNORED_BENCHMARKS_REGEX` with `FLAKY_BENCHMARKS_REGEX` on
microbenchmarks.
- **TL;DR: `FLAKY_BENCHMARKS_REGEX` is used by internal tooling to avoid
gating on flaky microbenchmarks. `IGNORED_BENCHMARKS_REGEX` was a
workaround.**
- `FLAKY_BENCHMARKS_REGEX` are excluded from gating decisions by
`benchmark_analyzer`.
- `FLAKY_BENCHMARKS_REGEX` is set on the `run-benchmarks` job, where
`benchmark_analyzer` convert marks flaky benchmarks. Downstream jobs
(`check-big-regressions`) read the flaky flag from the converted
results.
- Remove `check-big-regressions-no-ignored-benchmarks` (no longer
needed).

## Reason for change

[APMSP-2291 Identify and fix flaky dd-trace-dotnet microbenchmark
scenarios](https://datadoghq.atlassian.net/browse/APMSP-2291)

[APMSP-2292 Setup performance quality gates on
dd-trace-dotnet](https://datadoghq.atlassian.net/browse/APMSP-2292)

With #8300 reducing microbenchmark flakiness and with the ability to set
`FLAKY_BENCHMARKS_REGEX` to make `benchmark_analyzer` ignore flaky
benchmarks, we can now enforce the gates without false-positive noise
for microbenchmarks.

## Implementation details

- Flaky microbenchmarks on `FLAKY_BENCHMARKS_REGEX` were determined by
stability tests. They are summarized on #8300. I also added other
benchmarks @NachoEchevarria and I had classified as flaky before.
- Removed the shell script logic that renamed ignored benchmarks with
the `ignored.` prefix, `benchmark_analyzer` now handles this via
`FLAKY_BENCHMARKS_REGEX`.

## Test coverage

The most important bit to verify is whether we'd have flaky gates on
micro/macrobenchmarks.

We won't. Because:
- On microbenchmarks, the regression check threshold is 20%, which is
way bigger than variation for stable benchmarks.
- Our stability tests (see
#8300) show that
variation for stable benchmarks stay mostly under 20%. For flaky
benchmarks, this variation can be higher, but we ignore flaky benchmarks
by filtering them on `FLAKY_BENCHMARKS_REGEX`.
- On macrobenchmarks, SLO checks on macrobenchmarks are all passing for
a month without issue:
<img width="697" height="409" alt="image"
src="https://github.com/user-attachments/assets/97793111-3744-4e2a-9720-04e8ffade07a"
/>

## Other details

As a reminder, if bypassing is necessary, see
https://datadoghq.atlassian.net/wiki/x/8YFzMwE
andrewlock pushed a commit that referenced this pull request Apr 21, 2026
## Summary of changes

- Set `DD_CIVISIBILITY_AGENTLESS_ENABLED=1` on the benchmark process in
`.gitlab/benchmarks/microbenchmarks/scripts/run-benchmarks.ps1` when
`DD_API_KEY` is present, and default `DD_SITE=datadoghq.com`.
- Change the SSM parameter name in
`.gitlab/benchmarks/microbenchmarks.yml` from
`ci.dd-trace-dotnet.dd_api_key` to `ci.dd-trace-dotnet.dd_api_key-prod`
to match the key the previous pipeline was using.

## Reason for change

Microbenchmark results stopped being reported to Datadog on April 8,
2026, after #8300 moved the Windows microbenchmark runner from
`benchmarking-platform@dd-trace-dotnet/micro` into this repo. The
`Datadog.Trace.BenchmarkDotNet` exporter ships results to Datadog via CI
Visibility, and the old flow enabled agentless mode
(`DD_CIVISIBILITY_AGENTLESS_ENABLED=1`) with a production SSM key — two
details that were not carried over in the rewrite, so the exporter
silently failed to reach intake.

Cross-reference with the old script:
`benchmarking-platform/run-benchmarks.ps1:128`
(`$env:DD_CIVISIBILITY_AGENTLESS_ENABLED=1`) and
`benchmarking-platform/steps/run-windows-benchmarks.sh:9`
(`ci.${CI_PROJECT_NAME}.dd_api_key-prod`).

## Implementation details

- `run-benchmarks.ps1` now guards the agentless toggle on `DD_API_KEY`
being set, so local/dev invocations without a key continue to behave as
before (warn and skip agentless) rather than attempting — and failing —
to hit intake.
- No changes to `Build.cs`, tracer code, or the benchmark projects. This
is purely a CI config fix.

## Test coverage

- Verified the execution order in `bp-runner.windows.yml`: the env vars
are set in `run-benchmarks.ps1` after
`BuildTracerHome`/`BuildBenchmarks` and before the BDN CLI is invoked —
i.e., inherited by every BDN child process.

## Other details
igoragoli added a commit that referenced this pull request May 7, 2026
## Summary of changes

- Enable OpenTelemetry benchmarks which were turned off on
#8300.
- Use 1 CPU per benchmark instead of 2 to fit OpenTelemetry benchmarks.
- Reduce `launchCount` from 10 to 5.
    - Flakiness changes.
- This increases flakiness overall because of execution time/throughput
metrics.
            - The false positive (FP) rate went from ~4% to 9%.
        **- Memory allocation metrics remain stable.**
- The FP rate considering allocation metrics is 6%. There are some big
outliers¹, but 85% of the benchmarks have a 0% FP rate. By marking them
as flaky, we can have very effective gates on the remaining benchmarks.
    - Absolute value changes.
- This might lead to changes in absolute values on certain benchmarks
(mean delta of execution time and throughput is -11%² and -6%).
- **But once again, memory allocation metrics remain stable (mean delta
is of 0.1%).**
- This reduced CI job time in 10 minutes (from ~70 minutes to ~60
minutes). I was expecting a bigger change.
            - This might be because on 1 CPU the benchmarks are slower?
- Maybe reverting to 10 launches and having stabler benchmarks would be
better. But if we only care about memory allocation metrics, it is
already stable. And we send half of the data over the wire to the BP UI,
which leads to less curl retries.

<details>
<summary>¹ Worst offenders when it comes to allocated memory false
positives</summary>

| Scenario | FP Rate | Max Bound |
|----------|---------|-----------|
| StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1 |
**68.9%** | 14.69% |
| OTel Api ActivityBenchmark.StartSpan_GetContext_Sampled net6.0 | 55.6%
| 5.93% |
| OTel Api ActivityBenchmark.StartSpan_UpdateName_Sampled net6.0 | 55.6%
| 5.14% |
| OTel Api TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net6.0 |
55.6% | 6.01% |
| OTel Api ActivityBenchmark.StartSpan_AddEvent_Sampled net6.0 | 53.3% |
4.90% |
| OTel Api ActivityBenchmark.StartSpan_SetAttributes_Sampled net6.0 |
53.3% | 3.28% |
| OTel Api ActivityBenchmark.StartSpan_SetStatus_Sampled net6.0 | 53.3%
| 5.93% |
| OTel Api TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net6.0 |
53.3% | 5.93% |
| OTel Api TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net6.0 |
53.3% | 8.41% |
| OTel Api TracerBenchmark.StartActiveSpan net6.0 | 53.3% | 6.90% |
| OTel Api TracerBenchmark.StartRootSpan net6.0 | 53.3% | 8.41% |
| OTel Api TracerBenchmark.StartSpan_GetCurrentSpan net6.0 | 53.3% |
6.90% |

</details>

² I have no idea why both execution time and throughput went down. I
would imagine they would have inverse behaviors? Maybe this is because
of some BenchmarkDotNet internal? Anyways, execution time and throughput
are not looking super reliable for gating in any case.

## Reason for change

Restore OpenTelemetry benchmarks missing after bp-runner migration, and
improve CI job performance.

## Implementation details

## Test coverage

- Ran stability tests for all benchmarks after changes.
- If necessary, we can go into details on the PR conversation. But the
main conclusions are listed on the "Summary of changes" section.
- Results for this branch on the CIApp APM .NET Benchmarks dashboard:
https://app.datadoghq.com/dashboard/9eu-mfe-5ad?fromUser=true&refresh_mode=paused&tpl_var_branch%5B0%5D=%22augusto%2Fenable-otel-benchmarks%22&tpl_var_runtime%5B0%5D=%2A&from_ts=1777887699376&to_ts=1778159028000&live=false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:benchmarks Benchmarks, throughput tests, Crank, Bombardier, etc type:flake-fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants