Skip to content

[Dynamic Instrumentation] DEBUG-2842 Snapshot exploration test#6064

Closed
dudikeleti wants to merge 33 commits into
masterfrom
dudik/explorationtests-snapshots
Closed

[Dynamic Instrumentation] DEBUG-2842 Snapshot exploration test#6064
dudikeleti wants to merge 33 commits into
masterfrom
dudik/explorationtests-snapshots

Conversation

@dudikeleti

@dudikeleti dudikeleti commented Sep 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary of changes

This PR expand the curent exploration test functionality to test not only the produced bytcode but also the snapshot of a probe.

Reason for change

In many cases the instrumentation itself is valid but still we fail to create a valid snapshot for a certain probe.

Implementation details

We run open source projects tests and put probe on each method, then validate the produced snapshot.

Test coverage

Exploration tests

@datadog-ddstaging

datadog-ddstaging Bot commented Sep 23, 2024 •
edited
Loading

Copy link
Copy Markdown

Datadog Report

Branch report: dudik/explorationtests-snapshots
Commit report: 816321d
Test service: dd-trace-dotnet

❌ 31 Failed (0 Known Flaky), 441959 Passed, 3655 Skipped, 32h 18m 3.45s Total Time
❄️ 31 New Flaky
⌛ 1 Performance Regression

❌ Failed Tests (31)

This report shows up to 5 failed tests.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
     Expected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_ENABLED", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_PROBES_PATH", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_REPORT_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
     Expected keysWithoutConfig.OrderBy(x => x) to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_ENABLED", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_PROBES_PATH", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_REPORT_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
     Expected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_ENABLED", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_PROBES_PATH", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_REPORT_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
     Expected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_ENABLED", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_PROBES_PATH", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_REPORT_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
     Expected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_ENABLED", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_PROBES_PATH", "DD_INTERNAL_SNAPSHOT_EXPLORATION_TEST_REPORT_PATH"}.

New Flaky Tests (31)

  • TestConditions - Datadog.Trace.Tests.Debugger.DebuggerExpressionLanguageTests

  • TestConditions - Datadog.Trace.Tests.Debugger.DebuggerExpressionLanguageTests

  • TestConditions - Datadog.Trace.Tests.Debugger.DebuggerExpressionLanguageTests - Last Failure

    Expand for error 
     
 Verified path: D:\a\1\s\tracer\test\Datadog.Trace.Tests\Debugger\ProbeExpressionsResources/Approvals\DebuggerExpressionLanguageTests.GreaterThanStringChar.received.txt"getmember": [
           {
             "ref": "this"
           },
           "Char"
         ]
       }
     ]
   }
 ...

  • TestConditions - Datadog.Trace.Tests.Debugger.DebuggerExpressionLanguageTests - Last Failure

    Expand for error 
     Results do not match.
 Differences:
 Received: DebuggerExpressionLanguageTests.IllegalBinaryOperation.received.txt
 Verified: DebuggerExpressionLanguageTests.IllegalBinaryOperation.verified.txt
 Received Content:
 Condition:
 Json:
 {
     "gt": [
         "# SomeLocal should have a 'ref' prefix",
 ...

  • TestConditions - Datadog.Trace.Tests.Debugger.DebuggerExpressionLanguageTests

⌛ Performance Regressions vs Default Branch (1)

  • Profiler_garbagecollections_cpu_walltime - scenarios 2.75s (+205.72ms, +8%) - Details

@andrewlock

andrewlock commented Sep 23, 2024 •
edited by dd-trace-dotnet-ci-bot Bot
Loading

Copy link
Copy Markdown
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (6064) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration72.87 ± (72.75 - 73.00) ms72.80 ± (72.77 - 73.06) ms-0.1%✅
.NET Framework 4.8 - Bailout
duration76.75 ± (76.62 - 77.02) ms76.91 ± (76.75 - 77.16) ms+0.2%✅⬆️
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1071.84 ± (1071.48 - 1079.10) ms1082.32 ± (1080.77 - 1085.82) ms+1.0%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms22.70 ± (22.66 - 22.75) ms22.72 ± (22.66 - 22.77) ms+0.1%✅⬆️
process.time_to_main_ms85.49 ± (85.29 - 85.68) ms85.95 ± (85.69 - 86.20) ms+0.5%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.87 ± (10.86 - 10.88) MB10.91 ± (10.90 - 10.91) MB+0.3%✅⬆️
runtime.dotnet.threads.count12 ± (12 - 12)12 ± (12 - 12)+0.0%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms22.70 ± (22.66 - 22.74) ms22.43 ± (22.40 - 22.47) ms-1.2%✅
process.time_to_main_ms87.05 ± (86.82 - 87.28) ms85.74 ± (85.53 - 85.95) ms-1.5%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.94 ± (10.94 - 10.95) MB10.93 ± (10.93 - 10.94) MB-0.1%✅
runtime.dotnet.threads.count13 ± (13 - 13)13 ± (13 - 13)+0.0%✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms216.70 ± (215.63 - 217.76) ms189.75 ± (189.25 - 190.25) ms-12.4%✅
process.time_to_main_ms532.03 ± (530.49 - 533.57) ms531.97 ± (530.53 - 533.42) ms-0.0%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed47.87 ± (47.83 - 47.91) MB46.78 ± (46.73 - 46.83) MB-2.3%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.3%✅
.NET 6 - Baseline
process.internal_duration_ms21.29 ± (21.26 - 21.33) ms21.22 ± (21.17 - 21.27) ms-0.3%✅
process.time_to_main_ms73.53 ± (73.36 - 73.69) ms74.29 ± (74.05 - 74.54) ms+1.0%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.61 ± (10.60 - 10.61) MB10.64 ± (10.64 - 10.65) MB+0.4%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 6 - Bailout
process.internal_duration_ms21.64 ± (21.59 - 21.70) ms21.07 ± (21.04 - 21.11) ms-2.6%✅
process.time_to_main_ms77.22 ± (76.93 - 77.51) ms74.73 ± (74.54 - 74.92) ms-3.2%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed10.72 ± (10.72 - 10.73) MB10.76 ± (10.76 - 10.76) MB+0.4%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms381.21 ± (379.08 - 383.34) ms382.25 ± (378.99 - 385.52) ms+0.3%✅⬆️
process.time_to_main_ms527.99 ± (526.88 - 529.10) ms528.71 ± (527.55 - 529.88) ms+0.1%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed49.32 ± (49.29 - 49.35) MB48.39 ± (48.37 - 48.41) MB-1.9%✅
runtime.dotnet.threads.count28 ± (28 - 28)28 ± (28 - 28)-0.7%✅
.NET 8 - Baseline
process.internal_duration_ms19.61 ± (19.56 - 19.67) ms19.92 ± (19.86 - 19.98) ms+1.6%✅⬆️
process.time_to_main_ms73.80 ± (73.54 - 74.06) ms76.18 ± (75.87 - 76.49) ms+3.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.65 ± (7.65 - 7.66) MB7.68 ± (7.67 - 7.69) MB+0.4%✅⬆️
runtime.dotnet.threads.count10 ± (10 - 10)10 ± (10 - 10)+0.0%✅
.NET 8 - Bailout
process.internal_duration_ms19.37 ± (19.34 - 19.41) ms19.62 ± (19.58 - 19.65) ms+1.2%✅⬆️
process.time_to_main_ms73.78 ± (73.61 - 73.95) ms74.66 ± (74.48 - 74.83) ms+1.2%✅⬆️
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed7.70 ± (7.69 - 7.70) MB7.72 ± (7.72 - 7.73) MB+0.3%✅⬆️
runtime.dotnet.threads.count11 ± (11 - 11)11 ± (11 - 11)+0.0%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms302.83 ± (300.47 - 305.19) ms307.93 ± (305.66 - 310.20) ms+1.7%✅⬆️
process.time_to_main_ms493.82 ± (492.63 - 495.01) ms488.57 ± (487.04 - 490.10) ms-1.1%✅
runtime.dotnet.exceptions.count0 ± (0 - 0)0 ± (0 - 0)+0.0%✅
runtime.dotnet.mem.committed36.61 ± (36.59 - 36.63) MB35.98 ± (35.96 - 36.01) MB-1.7%✅
runtime.dotnet.threads.count27 ± (27 - 27)27 ± (27 - 27)+0.3%✅⬆️

HttpMessageHandler

Metric Master (Mean ± 95% CI) Current (Mean ± 95% CI) Change Status
.NET Framework 4.8 - Baseline
duration202.89 ± (203.48 - 204.96) ms202.96 ± (203.12 - 204.15) ms+0.0%✅⬆️
.NET Framework 4.8 - Bailout
duration207.57 ± (208.21 - 209.82) ms207.41 ± (207.15 - 208.17) ms-0.1%✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration1223.23 ± (1217.87 - 1226.62) ms1242.84 ± (1241.17 - 1246.33) ms+1.6%✅⬆️
.NET Core 3.1 - Baseline
process.internal_duration_ms198.90 ± (198.16 - 199.64) ms196.62 ± (196.05 - 197.19) ms-1.1%✅
process.time_to_main_ms86.17 ± (85.82 - 86.52) ms85.03 ± (84.77 - 85.29) ms-1.3%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.08 ± (16.06 - 16.09) MB15.99 ± (15.97 - 16.00) MB-0.5%✅
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)-1.2%✅
.NET Core 3.1 - Bailout
process.internal_duration_ms197.57 ± (196.91 - 198.23) ms195.61 ± (195.10 - 196.11) ms-1.0%✅
process.time_to_main_ms87.65 ± (87.36 - 87.95) ms86.60 ± (86.34 - 86.85) ms-1.2%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed16.05 ± (16.04 - 16.07) MB16.06 ± (16.04 - 16.08) MB+0.0%✅⬆️
runtime.dotnet.threads.count21 ± (21 - 21)21 ± (21 - 21)+0.8%✅⬆️
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms395.01 ± (393.47 - 396.55) ms382.44 ± (380.90 - 383.98) ms-3.2%✅
process.time_to_main_ms542.11 ± (540.17 - 544.05) ms534.48 ± (532.86 - 536.10) ms-1.4%✅
runtime.dotnet.exceptions.count3 ± (3 - 3)3 ± (3 - 3)+0.0%✅
runtime.dotnet.mem.committed58.67 ± (58.57 - 58.76) MB56.30 ± (56.16 - 56.44) MB-4.0%✅
runtime.dotnet.threads.count30 ± (30 - 30)30 ± (30 - 30)-1.1%✅
.NET 6 - Baseline
process.internal_duration_ms205.06 ± (204.30 - 205.81) ms202.52 ± (201.96 - 203.08) ms-1.2%✅
process.time_to_main_ms75.58 ± (75.26 - 75.90) ms74.60 ± (74.32 - 74.88) ms-1.3%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.25 ± (16.23 - 16.26) MB16.32 ± (16.30 - 16.34) MB+0.4%✅⬆️
runtime.dotnet.threads.count20 ± (19 - 20)19 ± (19 - 19)-1.9%✅
.NET 6 - Bailout
process.internal_duration_ms203.58 ± (202.83 - 204.33) ms203.30 ± (202.70 - 203.90) ms-0.1%✅
process.time_to_main_ms76.49 ± (76.17 - 76.80) ms75.80 ± (75.51 - 76.08) ms-0.9%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed16.27 ± (16.26 - 16.29) MB16.29 ± (16.27 - 16.31) MB+0.1%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)20 ± (20 - 20)-0.8%✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms594.68 ± (592.02 - 597.34) ms589.75 ± (585.15 - 594.35) ms-0.8%✅
process.time_to_main_ms543.61 ± (542.06 - 545.16) ms529.64 ± (528.47 - 530.82) ms-2.6%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed60.98 ± (60.89 - 61.08) MB59.99 ± (59.89 - 60.09) MB-1.6%✅
runtime.dotnet.threads.count31 ± (31 - 31)31 ± (31 - 31)+0.3%✅⬆️
.NET 8 - Baseline
process.internal_duration_ms202.07 ± (201.31 - 202.83) ms199.54 ± (198.94 - 200.14) ms-1.3%✅
process.time_to_main_ms74.26 ± (73.95 - 74.57) ms73.65 ± (73.33 - 73.97) ms-0.8%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.55 ± (11.54 - 11.57) MB11.69 ± (11.67 - 11.70) MB+1.2%✅⬆️
runtime.dotnet.threads.count19 ± (19 - 19)18 ± (18 - 19)-1.0%✅
.NET 8 - Bailout
process.internal_duration_ms202.16 ± (201.35 - 202.96) ms197.36 ± (196.79 - 197.93) ms-2.4%✅
process.time_to_main_ms75.87 ± (75.59 - 76.15) ms73.78 ± (73.59 - 73.97) ms-2.8%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed11.59 ± (11.58 - 11.61) MB11.74 ± (11.72 - 11.75) MB+1.3%✅⬆️
runtime.dotnet.threads.count20 ± (20 - 20)19 ± (19 - 19)-1.8%✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms518.03 ± (514.08 - 521.99) ms527.15 ± (523.84 - 530.45) ms+1.8%✅⬆️
process.time_to_main_ms498.87 ± (497.75 - 500.00) ms488.89 ± (487.67 - 490.11) ms-2.0%✅
runtime.dotnet.exceptions.count4 ± (4 - 4)4 ± (4 - 4)+0.0%✅
runtime.dotnet.mem.committed50.18 ± (50.14 - 50.22) MB49.74 ± (49.70 - 49.77) MB-0.9%✅
runtime.dotnet.threads.count30 ± (30 - 30)29 ± (29 - 30)-1.4%✅
Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts
FakeDbCommand (.NET Framework 4.8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (73ms)  : 71, 75
    master - mean (73ms)  : 71, 75

    section Bailout
    This PR (6064) - mean (77ms)  : 75, 79
    master - mean (77ms)  : 75, 79

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (1,083ms)  : 1054, 1113
    master - mean (1,075ms)  : 1017, 1133
Loading
FakeDbCommand (.NET Core 3.1) 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (116ms)  : 111, 121
    master - mean (115ms)  : 111, 119

    section Bailout
    This PR (6064) - mean (115ms)  : 112, 117
    master - mean (117ms)  : 114, 119

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (754ms)  : 719, 789
    master - mean (788ms)  : 765, 811
Loading
FakeDbCommand (.NET 6) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (102ms)  : 97, 107
    master - mean (101ms)  : 98, 104

    section Bailout
    This PR (6064) - mean (102ms)  : 99, 105
    master - mean (105ms)  : 100, 110

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (937ms)  : 881, 993
    master - mean (938ms)  : 904, 972
Loading
FakeDbCommand (.NET 8) 
gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (104ms)  : 99, 110
    master - mean (101ms)  : 96, 107

    section Bailout
    This PR (6064) - mean (102ms)  : 99, 105
    master - mean (101ms)  : 98, 103

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (829ms)  : 787, 871
    master - mean (827ms)  : 776, 879
Loading
HttpMessageHandler (.NET Framework 4.8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (204ms)  : 196, 211
    master - mean (204ms)  : 194, 215

    section Bailout
    This PR (6064) - mean (208ms)  : 200, 215
    master - mean (209ms)  : 197, 221

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (1,244ms)  : 1218, 1269
    master - mean (1,222ms)  : 1157, 1288
Loading
HttpMessageHandler (.NET Core 3.1) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (292ms)  : 280, 303
    master - mean (295ms)  : 280, 311

    section Bailout
    This PR (6064) - mean (292ms)  : 282, 303
    master - mean (297ms)  : 279, 315

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (955ms)  : 929, 981
    master - mean (978ms)  : 950, 1006
Loading
HttpMessageHandler (.NET 6) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (287ms)  : 275, 299
    master - mean (292ms)  : 273, 311

    section Bailout
    This PR (6064) - mean (289ms)  : 277, 302
    master - mean (290ms)  : 272, 309

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (1,147ms)  : 1073, 1221
    master - mean (1,169ms)  : 1123, 1216
Loading
HttpMessageHandler (.NET 8) 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (6064) - mean (283ms)  : 270, 296
    master - mean (290ms)  : 268, 313

    section Bailout
    This PR (6064) - mean (282ms)  : 272, 292
    master - mean (290ms)  : 272, 308

    section CallTarget+Inlining+NGEN
    This PR (6064) - mean (1,048ms)  : 999, 1096
    master - mean (1,053ms)  : 977, 1129
Loading

@andrewlock

andrewlock commented Sep 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6064) (11.340M)   : 0, 11339527
    master (11.262M)   : 0, 11261987
    benchmarks/2.9.0 (11.033M)   : 0, 11032866

    section Automatic
    This PR (6064) (7.364M)   : 0, 7364342
    master (7.252M)   : 0, 7252121
    benchmarks/2.9.0 (7.786M)   : 0, 7785853

    section Trace stats
    master (7.492M)   : 0, 7491512

    section Manual
    master (11.082M)   : 0, 11081514

    section Manual + Automatic
    This PR (6064) (6.707M)   : 0, 6707358
    master (6.511M)   : 0, 6511339

    section DD_TRACE_ENABLED=0
    master (10.305M)   : 0, 10305041
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6064) (9.575M)   : 0, 9574994
    master (9.633M)   : 0, 9633274
    benchmarks/2.9.0 (9.495M)   : 0, 9494821

    section Automatic
    This PR (6064) (6.472M)   : 0, 6471666
    master (6.436M)   : 0, 6436394

    section Trace stats
    master (6.676M)   : 0, 6676124

    section Manual
    master (9.575M)   : 0, 9575331

    section Manual + Automatic
    This PR (6064) (5.874M)   : 0, 5874057
    master (5.718M)   : 0, 5718155

    section DD_TRACE_ENABLED=0
    master (8.573M)   : 0, 8572716
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6064) (9.323M)   : 0, 9322956
    master (9.876M)   : 0, 9876001
    benchmarks/2.9.0 (10.020M)   : 0, 10019592

    section Automatic
    This PR (6064) (6.121M)   : crit ,0, 6121452
    master (6.512M)   : 0, 6511943
    benchmarks/2.9.0 (7.255M)   : 0, 7255257

    section Trace stats
    master (7.089M)   : 0, 7089152

    section Manual
    master (10.204M)   : 0, 10203572

    section Manual + Automatic
    This PR (6064) (5.786M)   : 0, 5786495
    master (6.050M)   : 0, 6050240

    section DD_TRACE_ENABLED=0
    master (9.471M)   : 0, 9471285
Loading

@andrewlock

andrewlock commented Sep 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

Benchmarks Report for tracer 🐌

Benchmarks for #6064 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.133
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.96μs 44.7ns 313ns 0.0202 0.00809 0 5.62 KB
master StartStopWithChild netcoreapp3.1 10.1μs 55.5ns 356ns 0.024 0.0144 0.00479 5.8 KB
master StartStopWithChild net472 16.2μs 56.4ns 219ns 1.04 0.299 0.0968 6.21 KB
#6064 StartStopWithChild net6.0 8μs 44.9ns 308ns 0.0161 0.00803 0 5.61 KB
#6064 StartStopWithChild netcoreapp3.1 9.9μs 47.5ns 184ns 0.0192 0.00958 0 5.8 KB
#6064 StartStopWithChild net472 16.1μs 71.6ns 277ns 1.04 0.299 0.102 6.21 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 499μs 260ns 937ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 649μs 400ns 1.55μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 843μs 573ns 2.22μs 0.419 0 0 3.3 KB
#6064 WriteAndFlushEnrichedTraces net6.0 495μs 478ns 1.85μs 0 0 0 2.7 KB
#6064 WriteAndFlushEnrichedTraces netcoreapp3.1 674μs 730ns 2.83μs 0 0 0 2.7 KB
#6064 WriteAndFlushEnrichedTraces net472 842μs 625ns 2.42μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 130μs 578ns 2.24μs 0.198 0 0 14.47 KB
master SendRequest netcoreapp3.1 150μs 234ns 907ns 0.223 0 0 17.27 KB
master SendRequest net472 0.000986ns 0.000254ns 0.000949ns 0 0 0 0 b
#6064 SendRequest net6.0 130μs 485ns 1.88μs 0.2 0 0 14.47 KB
#6064 SendRequest netcoreapp3.1 148μs 155ns 602ns 0.147 0 0 17.27 KB
#6064 SendRequest net472 0.000545ns 0.000269ns 0.00097ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 570μs 2.74μs 11μs 0.568 0 0 41.68 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 680μs 3.77μs 24.7μs 0.347 0 0 41.62 KB
master WriteAndFlushEnrichedTraces net472 851μs 4.17μs 17.7μs 8.22 2.47 0.411 53.32 KB
#6064 WriteAndFlushEnrichedTraces net6.0 585μs 3.28μs 20.5μs 0.568 0 0 41.57 KB
#6064 WriteAndFlushEnrichedTraces netcoreapp3.1 738μs 4.24μs 33.4μs 0.37 0 0 41.83 KB
#6064 WriteAndFlushEnrichedTraces net472 894μs 4.23μs 16.4μs 8.25 2.6 0.434 53.28 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.43μs 1.54ns 5.95ns 0.0142 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.77μs 2.3ns 8.91ns 0.0133 0 0 1.02 KB
master ExecuteNonQuery net472 2.1μs 1.62ns 5.85ns 0.156 0.00107 0 987 B
#6064 ExecuteNonQuery net6.0 1.31μs 1.15ns 4.45ns 0.0144 0 0 1.02 KB
#6064 ExecuteNonQuery netcoreapp3.1 1.78μs 3.96ns 14.8ns 0.0138 0 0 1.02 KB
#6064 ExecuteNonQuery net472 2.03μs 2.47ns 9.56ns 0.157 0.00102 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.25μs 0.455ns 1.76ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.58μs 4.69ns 18.2ns 0.0134 0 0 976 B
master CallElasticsearch net472 2.73μs 1.3ns 4.87ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.4μs 0.516ns 2ns 0.0133 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.62μs 1.27ns 4.58ns 0.0139 0 0 1.02 KB
master CallElasticsearchAsync net472 2.68μs 1.27ns 4.92ns 0.167 0 0 1.05 KB
#6064 CallElasticsearch net6.0 1.22μs 0.309ns 1.12ns 0.0136 0 0 976 B
#6064 CallElasticsearch netcoreapp3.1 1.56μs 0.798ns 2.99ns 0.0133 0 0 976 B
#6064 CallElasticsearch net472 2.68μs 1.77ns 6.85ns 0.157 0 0 995 B
#6064 CallElasticsearchAsync net6.0 1.32μs 1.13ns 4.37ns 0.0133 0 0 952 B
#6064 CallElasticsearchAsync netcoreapp3.1 1.64μs 0.919ns 3.44ns 0.014 0 0 1.02 KB
#6064 CallElasticsearchAsync net472 2.66μs 1.51ns 5.64ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.35μs 1.29ns 4.83ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.67μs 0.933ns 3.36ns 0.0125 0 0 952 B
master ExecuteAsync net472 1.76μs 0.378ns 1.31ns 0.145 0 0 915 B
#6064 ExecuteAsync net6.0 1.37μs 0.433ns 1.62ns 0.0132 0 0 952 B
#6064 ExecuteAsync netcoreapp3.1 1.67μs 1.65ns 6.19ns 0.0125 0 0 952 B
#6064 ExecuteAsync net472 1.75μs 0.397ns 1.54ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.41μs 1.19ns 4.6ns 0.033 0 0 2.31 KB
master SendAsync netcoreapp3.1 5.34μs 2.59ns 10ns 0.0376 0 0 2.85 KB
master SendAsync net472 7.51μs 1.99ns 7.45ns 0.495 0 0 3.12 KB
#6064 SendAsync net6.0 4.42μs 0.95ns 3.55ns 0.031 0 0 2.31 KB
#6064 SendAsync netcoreapp3.1 5.4μs 2.91ns 11.3ns 0.0379 0 0 2.85 KB
#6064 SendAsync net472 7.45μs 3.26ns 12.6ns 0.493 0 0 3.12 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.62μs 1.11ns 4.16ns 0.0233 0 0 1.69 KB
master EnrichedLog netcoreapp3.1 2.34μs 0.923ns 3.33ns 0.0223 0 0 1.69 KB
master EnrichedLog net472 2.67μs 1.21ns 4.52ns 0.258 0 0 1.62 KB
#6064 EnrichedLog net6.0 1.57μs 0.909ns 3.28ns 0.0235 0 0 1.69 KB
#6064 EnrichedLog netcoreapp3.1 2.24μs 0.915ns 3.42ns 0.0224 0 0 1.69 KB
#6064 EnrichedLog net472 2.61μs 1.36ns 5.08ns 0.258 0 0 1.62 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 111μs 164ns 636ns 0 0 0 4.3 KB
master EnrichedLog netcoreapp3.1 118μs 267ns 1.04μs 0 0 0 4.3 KB
master EnrichedLog net472 151μs 135ns 523ns 0.679 0.226 0 4.49 KB
#6064 EnrichedLog net6.0 112μs 125ns 483ns 0.0557 0 0 4.3 KB
#6064 EnrichedLog netcoreapp3.1 117μs 184ns 663ns 0 0 0 4.3 KB
#6064 EnrichedLog net472 150μs 121ns 468ns 0.679 0.226 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.07μs 1.06ns 3.67ns 0.0318 0 0 2.25 KB
master EnrichedLog netcoreapp3.1 4.19μs 3.78ns 14.6ns 0.0293 0 0 2.25 KB
master EnrichedLog net472 4.94μs 2.56ns 9.93ns 0.328 0 0 2.07 KB
#6064 EnrichedLog net6.0 3.06μs 1.12ns 4.33ns 0.0307 0 0 2.25 KB
#6064 EnrichedLog netcoreapp3.1 4.26μs 1.61ns 6.24ns 0.0298 0 0 2.25 KB
#6064 EnrichedLog net472 5μs 2.76ns 9.95ns 0.33 0 0 2.07 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.34μs 1.76ns 6.84ns 0.0161 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.79μs 0.716ns 2.77ns 0.0151 0 0 1.14 KB
master SendReceive net472 2.08μs 1.31ns 4.88ns 0.184 0 0 1.16 KB
#6064 SendReceive net6.0 1.3μs 0.532ns 2.06ns 0.0163 0 0 1.14 KB
#6064 SendReceive netcoreapp3.1 1.9μs 1.18ns 4.55ns 0.0152 0 0 1.14 KB
#6064 SendReceive net472 2.17μs 4.59ns 17.8ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.67μs 0.706ns 2.64ns 0.0226 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 3.96μs 1.49ns 5.75ns 0.0218 0 0 1.67 KB
master EnrichedLog net472 4.37μs 5.01ns 19.4ns 0.328 0 0 2.07 KB
#6064 EnrichedLog net6.0 2.79μs 0.804ns 2.9ns 0.0224 0 0 1.62 KB
#6064 EnrichedLog netcoreapp3.1 4.05μs 1.36ns 5.08ns 0.0223 0 0 1.67 KB
#6064 EnrichedLog net472 4.52μs 1.79ns 6.7ns 0.327 0 0 2.07 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6064

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.130 718.07 811.60

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 429ns 0.18ns 0.673ns 0.00814 0 0 576 B
master StartFinishSpan netcoreapp3.1 554ns 0.301ns 1.17ns 0.0078 0 0 576 B
master StartFinishSpan net472 642ns 0.2ns 0.748ns 0.0917 0 0 578 B
master StartFinishScope net6.0 494ns 0.122ns 0.473ns 0.00981 0 0 696 B
master StartFinishScope netcoreapp3.1 718ns 0.573ns 2.22ns 0.00921 0 0 696 B
master StartFinishScope net472 815ns 0.307ns 1.19ns 0.105 0 0 658 B
#6064 StartFinishSpan net6.0 407ns 0.17ns 0.637ns 0.00815 0 0 576 B
#6064 StartFinishSpan netcoreapp3.1 574ns 0.356ns 1.33ns 0.00786 0 0 576 B
#6064 StartFinishSpan net472 606ns 0.343ns 1.33ns 0.0916 0 0 578 B
#6064 StartFinishScope net6.0 493ns 0.199ns 0.745ns 0.00987 0 0 696 B
#6064 StartFinishScope netcoreapp3.1 811ns 0.488ns 1.89ns 0.00936 0 0 696 B
#6064 StartFinishScope net472 795ns 0.281ns 1.09ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6064

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.135 616.73 700.10

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 619ns 1.84ns 7.12ns 0.00988 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 956ns 0.501ns 1.94ns 0.0092 0 0 696 B
master RunOnMethodBegin net472 1.1μs 0.272ns 0.98ns 0.104 0 0 658 B
#6064 RunOnMethodBegin net6.0 700ns 0.272ns 1.05ns 0.00978 0 0 696 B
#6064 RunOnMethodBegin netcoreapp3.1 938ns 0.533ns 2.06ns 0.0094 0 0 696 B
#6064 RunOnMethodBegin net472 1.05μs 0.279ns 1.08ns 0.104 0 0 658 B

@dudikeleti
dudikeleti marked this pull request as ready for review September 26, 2024 12:25
@dudikeleti
dudikeleti requested review from a team as code owners September 26, 2024 12:25
@GreenMatan
GreenMatan marked this pull request as draft October 10, 2024 11:32
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch 4 times, most recently from 1d82022 to f154c5c Compare October 14, 2024 15:02
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch 2 times, most recently from 5b79941 to f412eb8 Compare October 21, 2024 13:16
@dudikeleti
dudikeleti marked this pull request as ready for review October 24, 2024 15:12
@dudikeleti
dudikeleti marked this pull request as draft November 11, 2024 12:29
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch 2 times, most recently from fc5b6b1 to c37f927 Compare December 10, 2024 10:57
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch from 816321d to a25c696 Compare March 17, 2025 12:53
@datadog-datadog-prod-us1

This comment has been minimized.

Sign in to view
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch 2 times, most recently from f8ddb56 to 38078bb Compare March 25, 2025 13:52
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch from 38078bb to f71a985 Compare November 26, 2025 14:59
@pr-commenter

pr-commenter Bot commented Nov 28, 2025 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-05-06 10:21:28

Comparing candidate commit f070f97 in PR branch dudik/explorationtests-snapshots with baseline commit a60cc68 in branch master.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 26 metrics, 0 unstable metrics, 58 known flaky benchmarks, 29 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:Benchmarks.Trace.HttpClientBenchmark.SendAsync net472

  • 🟥 throughput [-7604.925op/s; -6226.837op/s] or [-8.682%; -7.108%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0

  • 🟩 throughput [+8171.971op/s; +10415.490op/s] or [+6.869%; +8.755%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 execution_time [+316.020ms; +320.893ms] or [+156.820%; +159.238%]
  • 🟥 throughput [-43.915op/s; -40.029op/s] or [-7.901%; -7.202%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • 🟥 execution_time [+375.648ms; +377.432ms] or [+296.785%; +298.194%]
  • 🟩 throughput [+98.745op/s; +101.784op/s] or [+13.019%; +13.420%]

scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+394.509ms; +397.563ms] or [+349.125%; +351.828%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472

  • 🟥 allocated_mem [+1.308KB; +1.308KB] or [+27.529%; +27.541%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+9.977%; +9.987%]
  • 🟩 execution_time [-16.297ms; -12.117ms] or [-7.611%; -5.659%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+27.502%; +27.510%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472

  • 🟥 allocated_mem [+1.307KB; +1.307KB] or [+105.746%; +105.759%]
  • 🟥 throughput [-272085.047op/s; -268149.154op/s] or [-27.781%; -27.379%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0

  • 🟥 allocated_mem [+471 bytes; +472 bytes] or [+38.558%; +38.566%]
  • 🟩 execution_time [-25.771ms; -20.911ms] or [-11.493%; -9.326%]
  • 🟥 throughput [-75458.042op/s; -52474.809op/s] or [-8.061%; -5.606%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1

  • 🟥 allocated_mem [+1.272KB; +1.272KB] or [+105.292%; +105.304%]
  • 🟥 throughput [-142062.876op/s; -125159.924op/s] or [-20.412%; -17.983%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0

  • 🟩 throughput [+248114.690op/s; +264628.023op/s] or [+8.273%; +8.824%]

scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1

  • 🟩 execution_time [-18.874ms; -14.501ms] or [-8.700%; -6.685%]
  • 🟩 throughput [+181866.270op/s; +235862.861op/s] or [+7.219%; +9.362%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472

  • 🟥 execution_time [+299.539ms; +300.116ms] or [+149.669%; +149.958%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0

  • 🟥 execution_time [+299.325ms; +302.446ms] or [+150.950%; +152.524%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1

  • 🟥 execution_time [+300.081ms; +302.553ms] or [+151.157%; +152.403%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472

  • 🟥 execution_time [+297.116ms; +297.749ms] or [+145.931%; +146.243%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0

  • 🟥 execution_time [+297.377ms; +300.893ms] or [+145.377%; +147.096%]

scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1

  • 🟥 execution_time [+300.651ms; +301.750ms] or [+150.265%; +150.814%]

scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0

  • 🟥 execution_time [+21.698µs; +45.295µs] or [+6.927%; +14.460%]
  • 🟥 throughput [-422.528op/s; -223.963op/s] or [-13.171%; -6.982%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472

  • 🟥 execution_time [+299.639ms; +300.335ms] or [+149.550%; +149.898%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

  • unstable execution_time [+361.376ms; +399.972ms] or [+392.650%; +434.586%]
  • 🟩 throughput [+915.759op/s; +1078.851op/s] or [+7.525%; +8.865%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

  • unstable execution_time [+267.881ms; +307.647ms] or [+203.399%; +233.593%]
  • 🟩 throughput [+653.094op/s; +855.700op/s] or [+6.322%; +8.284%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

  • 🟥 allocated_mem [+3.204KB; +3.209KB] or [+5.692%; +5.701%]
  • unstable execution_time [+276.854ms; +313.366ms] or [+127.295%; +144.083%]
  • 🟥 throughput [-513.787op/s; -479.430op/s] or [-46.554%; -43.441%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0

  • unstable execution_time [+174.861ms; +311.858ms] or [+74.518%; +132.901%]
  • 🟥 throughput [-666.740op/s; -583.286op/s] or [-44.472%; -38.905%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1

  • 🟥 execution_time [+330.154ms; +339.436ms] or [+197.471%; +203.022%]
  • 🟥 throughput [-388.449op/s; -353.724op/s] or [-27.047%; -24.629%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

  • 🟩 execution_time [-297.147µs; -145.801µs] or [-10.337%; -5.072%]
  • 🟩 throughput [+25.020op/s; +55.001op/s] or [+7.192%; +15.810%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0

  • 🟩 execution_time [-178.160µs; -146.444µs] or [-9.025%; -7.418%]
  • 🟩 throughput [+42.080op/s; +50.573op/s] or [+8.307%; +9.984%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1

  • unstable throughput [+14.911op/s; +40.675op/s] or [+5.880%; +16.039%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472

  • 🟥 execution_time [+301.454ms; +303.032ms] or [+151.807%; +152.601%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0

  • 🟥 execution_time [+302.105ms; +304.246ms] or [+151.385%; +152.458%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1

  • 🟥 execution_time [+300.495ms; +303.764ms] or [+150.956%; +152.598%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472

  • 🟥 execution_time [+303.535ms; +304.955ms] or [+152.425%; +153.138%]
  • 🟩 throughput [+18497.625op/s; +20205.953op/s] or [+6.197%; +6.769%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0

  • 🟥 execution_time [+298.511ms; +300.533ms] or [+147.600%; +148.600%]

scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1

  • 🟥 execution_time [+304.175ms; +307.792ms] or [+154.169%; +156.003%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472

  • 🟥 execution_time [+299.589ms; +301.397ms] or [+150.367%; +151.274%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0

  • 🟥 execution_time [+298.671ms; +300.710ms] or [+148.860%; +149.876%]
  • 🟩 throughput [+54017.481op/s; +58599.914op/s] or [+10.726%; +11.636%]

scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1

  • 🟥 execution_time [+301.087ms; +303.637ms] or [+149.788%; +151.057%]

scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0

  • 🟩 execution_time [-16.848ms; -13.190ms] or [-7.834%; -6.133%]
  • 🟩 throughput [+19018.456op/s; +25741.473op/s] or [+5.217%; +7.062%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472

  • unstable execution_time [+4.931µs; +45.580µs] or [+1.218%; +11.259%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0

  • 🟩 allocated_mem [-18.920KB; -18.899KB] or [-6.901%; -6.894%]
  • unstable execution_time [-47.114µs; +5.048µs] or [-9.312%; +0.998%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0

  • 🟥 execution_time [+5.727µs; +9.416µs] or [+13.537%; +22.255%]
  • 🟥 throughput [-4463.431op/s; -2764.969op/s] or [-18.790%; -11.640%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

  • unstable execution_time [-13.837µs; -6.493µs] or [-21.468%; -10.074%]
  • 🟩 throughput [+1704.072op/s; +3247.814op/s] or [+10.455%; +19.926%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472

  • 🟥 execution_time [+301.432ms; +302.933ms] or [+152.361%; +153.119%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+301.779ms; +303.952ms] or [+153.605%; +154.710%]

scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+297.355ms; +299.775ms] or [+148.863%; +150.075%]

scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0

  • 🟩 throughput [+37590.098op/s; +41717.888op/s] or [+7.115%; +7.896%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472

  • 🟥 execution_time [+300.554ms; +302.096ms] or [+149.799%; +150.568%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0

  • 🟥 execution_time [+302.046ms; +303.476ms] or [+151.673%; +152.391%]

scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1

  • 🟥 execution_time [+303.924ms; +306.127ms] or [+154.131%; +155.248%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472

  • 🟥 execution_time [+299.481ms; +300.342ms] or [+149.383%; +149.812%]
  • 🟩 throughput [+55551057.760op/s; +63394276.826op/s] or [+40.456%; +46.168%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0

  • 🟥 execution_time [+425.161ms; +429.082ms] or [+528.763%; +533.640%]
  • 🟩 throughput [+1089.376op/s; +1260.352op/s] or [+8.421%; +9.743%]

scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1

  • 🟥 execution_time [+299.771ms; +300.717ms] or [+149.519%; +149.991%]
  • 🟩 throughput [+18243506.960op/s; +19188488.182op/s] or [+8.081%; +8.499%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0

  • 🟩 throughput [+92784.473op/s; +100740.906op/s] or [+8.663%; +9.406%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0

  • 🟩 throughput [+89634.436op/s; +128718.340op/s] or [+6.938%; +9.963%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1

  • 🟩 throughput [+83915.060op/s; +91996.455op/s] or [+8.334%; +9.137%]

scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0

  • 🟩 throughput [+34651.484op/s; +42194.101op/s] or [+6.292%; +7.662%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472

  • 🟥 throughput [-44523.210op/s; -40961.749op/s] or [-6.516%; -5.995%]

scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0

  • 🟩 throughput [+71516.605op/s; +89543.275op/s] or [+7.990%; +10.004%]

Known flaky benchmarks without significant changes:

  • scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472
  • scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472
  • scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1
  • scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472
  • scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1
  • scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1
  • scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
  • scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472
  • scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
  • scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch 2 times, most recently from bd94d24 to a3555b5 Compare April 28, 2026 19:30
dudikeleti and others added 25 commits May 5, 2026 20:14
@dudikeleti
use static instance of NoOpSymbolsUploader
e07c827
@dudikeleti
create unique file per process
4db5e03
@dudikeleti
report error when there is an open generic type and the generic type …
243eb49 
…is a value type
@dudikeleti
add nullable enable
0c794db
@dudikeleti
Skip snapshot exploration test for predefined assemblies
bfac855
@dudikeleti
change csv format
0e6363f
@dudikeleti
fix csv format
1bba7ff
@dudikeleti
fix build
430a6f5
@dudikeleti
fix build
dad2499
@dudikeleti
change paths
9c6f0b8
@dudikeleti
enable adding probe processors in case of test
8557b7f
@dudikeleti
do not add multiple probes
01afc7c
@dudikeleti
read logs from native and wip
6364ff2
@dudikeleti
rebasing
31d3d6a
@dudikeleti
rebase wip
0819aeb
@dudikeleti
finish rebase
5b6e3ad
@dudikeleti
wip
8f84046
@dudikeleti
revert me
3f257c5
@dudikeleti
snapshot exploration test wip
c83128e
@dudikeleti
Stabilize snapshot exploration tests and simplify config
0999e43 
Fix rebase fallout in the snapshot exploration test path and make the harness reliable again. Share the exploration CSV sink between snapshot and log uploaders, disable hands-off config for isolated test hosts, and fail fast when probes are defined but none are installed so bootstrap problems are surfaced instead of passing silently.

Reduce the supported snapshot exploration configuration to ENABLED plus a single root path, derive the probes/report locations from that root, and keep extra tuning/debug switches out of the supported config surface. Clean up the snapshot-flow debug logging so it stays purely diagnostic and can be removed without affecting behavior.
@dudikeleti
clone pinned repos at the target tag
721d938
@dudikeleti
disable agent commumication in exploration test
7495873
@dudikeleti
fix testhost matching
9b8ae74
@dudikeleti
fix extraction of type and method
2b0da8e
@dudikeleti
fix how we collect values for nullable types
7d4fec8
@dudikeleti
dudikeleti force-pushed the dudik/explorationtests-snapshots branch from a3555b5 to 7d4fec8 Compare May 5, 2026 18:22
@dudikeleti @cursoragent
production-ready cleanup of snapshot exploration test branch
f070f97 
- isolate snapshot exploration test plumbing in DebuggerManager.SnapshotExploration.cs partial; revert Instrumentation.cs to master
- consolidate exploration-test enabled flags into ExplorationTestState
- extract testhost name and native log polling values to SnapshotExplorationConstants
- await DI initialization task instead of polling with Thread.Sleep
- remove SnapshotFlowDebugLog and its 12 hot-path call sites; restore master debug logs in LineDebuggerInvoker null-byref skips
- restore WriteCallsToLogArg at non-async ApplyMethodProbe exit to fix ref/out parameter capture
- delete unused top-level BoundLineProbeLocation
- set DD_TRACE_ENABLED=0 for snapshot scenarios to disable agent communication; restore VSTEST_CONNECTION_TIMEOUT=200 for non-snapshot runs
- minor fixes in Redaction (Nullable<T>), SnapshotPruner (bounds checks), ProbeExpressionParser (generic type arg substitution)

Co-authored-by: Cursor <[email protected]>
@dudikeleti dudikeleti mentioned this pull request May 6, 2026
Open
@dudikeleti dudikeleti closed this Jun 10, 2026
@dudikeleti

dudikeleti commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

see #8572

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area:debugger

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dudikeleti @andrewlock