Skip to content

List iteration benchmark#10888

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 15 commits into
masterfrom
dougqh/list-iteration-benchmark
Jun 24, 2026
Merged

List iteration benchmark#10888
gh-worker-dd-mergequeue-cf854d[bot] merged 15 commits into
masterfrom
dougqh/list-iteration-benchmark

Conversation

@dougqh

@dougqh dougqh commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Benchmark examining different ways to iterate over Lists: Collections.emptyList, Collections.singletonList, and ArrayList of different sizes

Covers multiple iteration options...

  • c-style for ( int i = 0; i < list.size(); ++i ) ...
  • iterator
  • enhanced for ( x : list ) (sugar for iterator)
  • List.forEach
  • List.stream
  • List.parallelStream

Each benchmark has a variation with and without the loop body fully inlined

Motivation

Illustrate overhead and make recommendations about the preferred way to iterate over lists in dd-java-agent.

Additional Notes

Recommends are...

  • prefer enhanced for -- except in the uncommon situation where escape analysis fails -- in that case, forEach or c-style are the best altenatives
  • avoid streams - always incur extra allocation - generally unnecessary for the task at hand

Contributor Checklist

Jira ticket: [PROJ-IDENT]

Note: Once your PR is ready to merge, add it to the merge queue by commenting /merge. /merge -c cancels the queue request. /merge -f --reason "reason" skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.

@dougqh
dougqh requested a review from a team as a code owner March 18, 2026 14:00
@dougqh
dougqh requested a review from ygree March 18, 2026 14:00
@dougqh dougqh added comp: core Tracer core type: documentation tag: performance Performance related changes labels Mar 18, 2026
@pr-commenter

pr-commenter Bot commented Mar 18, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

⚠️ Warning: Baseline build not found for merge-base commit. Comparing against the latest commit on master instead.

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/list-iteration-benchmark
git_commit_date 1782158081 1782171008
git_commit_sha ea827ba b816fde
release_version 1.64.0-SNAPSHOT~ea827ba0c4 1.61.0-SNAPSHOT~b816fdeed7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1782173150 1782173150
ci_job_id 1794044753 1794044753
ci_pipeline_id 120371156 120371156
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-oyjdvy3d 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-oyjdvy3d 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 8 performance improvements and 1 performance regressions! Performance is the same for 48 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast:BytebuddyAgent better
[-32.818ms; -19.041ms] or [-3.969%; -2.303%]		 800.992ms 826.921ms
scenario:startup:petclinic:appsec:BytebuddyAgent better
[-19.483ms; -13.711ms] or [-2.869%; -2.019%]		 662.427ms 679.024ms
scenario:startup:petclinic:appsec:AppSec better
[-5.755ms; -4.441ms] or [-3.042%; -2.347%]		 184.125ms 189.223ms
scenario:startup:petclinic:appsec:Telemetry worse
[+346.258µs; +791.768µs] or [+4.357%; +9.964%]		 8.516ms 7.947ms
scenario:startup:petclinic:appsec:Flare Poller better
[-5.904ms; -5.541ms] or [-63.776%; -59.855%]		 3.535ms 9.258ms
scenario:startup:petclinic:profiling:Agent.start better
[-152.079ms; -102.303ms] or [-11.456%; -7.706%]		 1.200s 1.328s
scenario:startup:petclinic:profiling:Remote Config better
[-69.886µs; -19.354µs] or [-11.263%; -3.119%]		 575.852µs 620.472µs
scenario:startup:petclinic:tracing:AgentMeter better
[-1081.498µs; -652.191µs] or [-3.588%; -2.164%]		 29.277ms 30.144ms
scenario:startup:petclinic:tracing:Telemetry better
[-519.928µs; -318.758µs] or [-6.133%; -3.760%]		 8.058ms 8.477ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.071 s) : 0, 1071462
Total [baseline] (10.964 s) : 0, 10963509
Agent [candidate] (1.059 s) : 0, 1058573
Total [candidate] (11.025 s) : 0, 11024502
section appsec
Agent [baseline] (1.274 s) : 0, 1273545
Total [baseline] (10.951 s) : 0, 10950504
Agent [candidate] (1.248 s) : 0, 1248302
Total [candidate] (11.196 s) : 0, 11195889
section iast
Agent [baseline] (1.256 s) : 0, 1256403
Total [baseline] (11.352 s) : 0, 11352370
Agent [candidate] (1.231 s) : 0, 1231317
Total [candidate] (11.294 s) : 0, 11294026
section profiling
Agent [baseline] (1.328 s) : 0, 1327526
Total [baseline] (11.145 s) : 0, 11144568
Agent [candidate] (1.2 s) : 0, 1200335
Total [candidate] (11.019 s) : 0, 11018783
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.071 s -
Agent appsec 1.274 s 202.083 ms (18.9%)
Agent iast 1.256 s 184.941 ms (17.3%)
Agent profiling 1.328 s 256.063 ms (23.9%)
Total tracing 10.964 s -
Total appsec 10.951 s -13.005 ms (-0.1%)
Total iast 11.352 s 388.861 ms (3.5%)
Total profiling 11.145 s 181.059 ms (1.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.059 s -
Agent appsec 1.248 s 189.729 ms (17.9%)
Agent iast 1.231 s 172.744 ms (16.3%)
Agent profiling 1.2 s 141.762 ms (13.4%)
Total tracing 11.025 s -
Total appsec 11.196 s 171.387 ms (1.6%)
Total iast 11.294 s 269.524 ms (2.4%)
Total profiling 11.019 s -5.718 ms (-0.1%) 
gantt
    title petclinic - break down per module: candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.22 ms) : 0, 1220
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (641.003 ms) : 0, 641003
BytebuddyAgent [candidate] (632.476 ms) : 0, 632476
AgentMeter [baseline] (30.144 ms) : 0, 30144
AgentMeter [candidate] (29.277 ms) : 0, 29277
GlobalTracer [baseline] (249.378 ms) : 0, 249378
GlobalTracer [candidate] (248.848 ms) : 0, 248848
AppSec [baseline] (32.293 ms) : 0, 32293
AppSec [candidate] (32.083 ms) : 0, 32083
Debugger [baseline] (60.37 ms) : 0, 60370
Debugger [candidate] (59.862 ms) : 0, 59862
Remote Config [baseline] (589.811 µs) : 0, 590
Remote Config [candidate] (600.01 µs) : 0, 600
Telemetry [baseline] (8.477 ms) : 0, 8477
Telemetry [candidate] (8.058 ms) : 0, 8058
Flare Poller [baseline] (11.208 ms) : 0, 11208
Flare Poller [candidate] (9.887 ms) : 0, 9887
section appsec
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.233 ms) : 0, 1233
BytebuddyAgent [baseline] (679.024 ms) : 0, 679024
BytebuddyAgent [candidate] (662.427 ms) : 0, 662427
AgentMeter [baseline] (12.143 ms) : 0, 12143
AgentMeter [candidate] (12.006 ms) : 0, 12006
GlobalTracer [baseline] (247.96 ms) : 0, 247960
GlobalTracer [candidate] (248.988 ms) : 0, 248988
AppSec [baseline] (189.223 ms) : 0, 189223
AppSec [candidate] (184.125 ms) : 0, 184125
Debugger [baseline] (65.148 ms) : 0, 65148
Debugger [candidate] (65.962 ms) : 0, 65962
Remote Config [baseline] (579.344 µs) : 0, 579
Remote Config [candidate] (599.113 µs) : 0, 599
Telemetry [baseline] (7.947 ms) : 0, 7947
Telemetry [candidate] (8.516 ms) : 0, 8516
Flare Poller [baseline] (9.258 ms) : 0, 9258
Flare Poller [candidate] (3.535 ms) : 0, 3535
IAST [baseline] (24.276 ms) : 0, 24276
IAST [candidate] (24.614 ms) : 0, 24614
section iast
crashtracking [baseline] (1.204 ms) : 0, 1204
crashtracking [candidate] (1.236 ms) : 0, 1236
BytebuddyAgent [baseline] (834.191 ms) : 0, 834191
BytebuddyAgent [candidate] (806.659 ms) : 0, 806659
AgentMeter [baseline] (11.651 ms) : 0, 11651
AgentMeter [candidate] (11.512 ms) : 0, 11512
GlobalTracer [baseline] (243.944 ms) : 0, 243944
GlobalTracer [candidate] (240.19 ms) : 0, 240190
AppSec [baseline] (25.854 ms) : 0, 25854
AppSec [candidate] (31.689 ms) : 0, 31689
Debugger [baseline] (64.122 ms) : 0, 64122
Debugger [candidate] (61.991 ms) : 0, 61991
Remote Config [baseline] (521.402 µs) : 0, 521
Remote Config [candidate] (532.639 µs) : 0, 533
Telemetry [baseline] (7.879 ms) : 0, 7879
Telemetry [candidate] (11.667 ms) : 0, 11667
Flare Poller [baseline] (3.323 ms) : 0, 3323
Flare Poller [candidate] (3.463 ms) : 0, 3463
IAST [baseline] (26.943 ms) : 0, 26943
IAST [candidate] (25.878 ms) : 0, 25878
section profiling
crashtracking [baseline] (549.919 µs) : 0, 550
crashtracking [candidate] (1.196 ms) : 0, 1196
BytebuddyAgent [baseline] (695.831 ms) : 0, 695831
BytebuddyAgent [candidate] (701.345 ms) : 0, 701345
AgentMeter [baseline] (9.943 ms) : 0, 9943
AgentMeter [candidate] (9.275 ms) : 0, 9275
GlobalTracer [baseline] (211.113 ms) : 0, 211113
GlobalTracer [candidate] (210.042 ms) : 0, 210042
AppSec [baseline] (32.742 ms) : 0, 32742
AppSec [candidate] (32.822 ms) : 0, 32822
Debugger [baseline] (66.641 ms) : 0, 66641
Debugger [candidate] (66.157 ms) : 0, 66157
Remote Config [baseline] (620.472 µs) : 0, 620
Remote Config [candidate] (575.852 µs) : 0, 576
Telemetry [baseline] (8.057 ms) : 0, 8057
Telemetry [candidate] (7.995 ms) : 0, 7995
Flare Poller [baseline] (3.597 ms) : 0, 3597
Flare Poller [candidate] (3.614 ms) : 0, 3614
ProfilingAgent [baseline] (96.023 ms) : 0, 96023
ProfilingAgent [candidate] (95.093 ms) : 0, 95093
Profiling [baseline] (96.583 ms) : 0, 96583
Profiling [candidate] (95.673 ms) : 0, 95673
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1066231
Total [baseline] (8.871 s) : 0, 8871250
Agent [candidate] (1.067 s) : 0, 1067439
Total [candidate] (8.855 s) : 0, 8855192
section iast
Agent [baseline] (1.246 s) : 0, 1246472
Total [baseline] (9.491 s) : 0, 9490893
Agent [candidate] (1.225 s) : 0, 1224996
Total [candidate] (9.576 s) : 0, 9576358
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.066 s -
Agent iast 1.246 s 180.24 ms (16.9%)
Total tracing 8.871 s -
Total iast 9.491 s 619.643 ms (7.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.067 s -
Agent iast 1.225 s 157.557 ms (14.8%)
Total tracing 8.855 s -
Total iast 9.576 s 721.166 ms (8.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.208 ms) : 0, 1208
crashtracking [candidate] (1.237 ms) : 0, 1237
BytebuddyAgent [baseline] (638.433 ms) : 0, 638433
BytebuddyAgent [candidate] (639.455 ms) : 0, 639455
AgentMeter [baseline] (29.851 ms) : 0, 29851
AgentMeter [candidate] (29.77 ms) : 0, 29770
GlobalTracer [baseline] (247.817 ms) : 0, 247817
GlobalTracer [candidate] (251.511 ms) : 0, 251511
AppSec [baseline] (32.266 ms) : 0, 32266
AppSec [candidate] (32.381 ms) : 0, 32381
Debugger [baseline] (59.64 ms) : 0, 59640
Debugger [candidate] (59.711 ms) : 0, 59711
Remote Config [baseline] (601.099 µs) : 0, 601
Remote Config [candidate] (605.219 µs) : 0, 605
Telemetry [baseline] (8.466 ms) : 0, 8466
Telemetry [candidate] (8.224 ms) : 0, 8224
Flare Poller [baseline] (11.22 ms) : 0, 11220
Flare Poller [candidate] (8.257 ms) : 0, 8257
section iast
crashtracking [baseline] (1.216 ms) : 0, 1216
crashtracking [candidate] (1.217 ms) : 0, 1217
BytebuddyAgent [baseline] (826.921 ms) : 0, 826921
BytebuddyAgent [candidate] (800.992 ms) : 0, 800992
AgentMeter [baseline] (11.483 ms) : 0, 11483
AgentMeter [candidate] (11.46 ms) : 0, 11460
GlobalTracer [baseline] (238.777 ms) : 0, 238777
GlobalTracer [candidate] (239.791 ms) : 0, 239791
AppSec [baseline] (28.063 ms) : 0, 28063
AppSec [candidate] (31.177 ms) : 0, 31177
Debugger [baseline] (63.063 ms) : 0, 63063
Debugger [candidate] (59.89 ms) : 0, 59890
Remote Config [baseline] (524.688 µs) : 0, 525
Remote Config [candidate] (1.145 ms) : 0, 1145
Telemetry [baseline] (7.753 ms) : 0, 7753
Telemetry [candidate] (13.668 ms) : 0, 13668
Flare Poller [baseline] (3.279 ms) : 0, 3279
Flare Poller [candidate] (3.656 ms) : 0, 3656
IAST [baseline] (28.679 ms) : 0, 28679
IAST [candidate] (25.851 ms) : 0, 25851
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/list-iteration-benchmark
git_commit_date 1782158081 1782171008
git_commit_sha ea827ba b816fde
release_version 1.64.0-SNAPSHOT~ea827ba0c4 1.61.0-SNAPSHOT~b816fdeed7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1782173769 1782173769
ci_job_id 1794044754 1794044754
ci_pipeline_id 120371156 120371156
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-4z3ubgzq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-4z3ubgzq 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 6 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast_FULL:high_load worse
[+331.345µs; +620.788µs] or [+6.179%; +11.576%]		 worse
[+0.675ms; +1.476ms] or [+5.263%; +11.508%]		 unstable
[-125.123op/s; +20.311op/s] or [-16.404%; +2.663%]		 5.839ms 13.898ms 710.344op/s 5.363ms 12.823ms 762.750op/s
scenario:load:petclinic:profiling:high_load worse
[+1.848ms; +2.159ms] or [+10.353%; +12.098%]		 worse
[+1.745ms; +2.565ms] or [+5.935%; +8.725%]		 unstable
[-46.751op/s; -0.561op/s] or [-18.318%; -0.220%]		 19.850ms 31.552ms 231.562op/s 17.847ms 29.398ms 255.219op/s
scenario:load:petclinic:iast:high_load worse
[+358.313µs; +1172.597µs] or [+2.028%; +6.638%]		 worse
[+0.912ms; +2.099ms] or [+3.208%; +7.381%]		 unstable
[-35.857op/s; +13.169op/s] or [-13.748%; +5.049%]		 18.430ms 29.944ms 249.469op/s 17.664ms 28.438ms 260.812op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.278 ms) : 1266, 1291
.   : milestone, 1278,
iast (3.355 ms) : 3303, 3407
.   : milestone, 3355,
iast_FULL (6.063 ms) : 6001, 6125
.   : milestone, 6063,
iast_GLOBAL (3.583 ms) : 3536, 3630
.   : milestone, 3583,
profiling (2.161 ms) : 2141, 2181
.   : milestone, 2161,
tracing (1.929 ms) : 1912, 1945
.   : milestone, 1929,
section candidate
no_agent (1.276 ms) : 1263, 1288
.   : milestone, 1276,
iast (3.327 ms) : 3280, 3374
.   : milestone, 3327,
iast_FULL (6.516 ms) : 6447, 6584
.   : milestone, 6516,
iast_GLOBAL (3.631 ms) : 3571, 3691
.   : milestone, 3631,
profiling (2.348 ms) : 2324, 2373
.   : milestone, 2348,
tracing (1.935 ms) : 1918, 1953
.   : milestone, 1935,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.278 ms [1.266 ms, 1.291 ms] -
iast 3.355 ms [3.303 ms, 3.407 ms] 2.077 ms (162.4%)
iast_FULL 6.063 ms [6.001 ms, 6.125 ms] 4.784 ms (374.2%)
iast_GLOBAL 3.583 ms [3.536 ms, 3.63 ms] 2.304 ms (180.2%)
profiling 2.161 ms [2.141 ms, 2.181 ms] 882.297 µs (69.0%)
tracing 1.929 ms [1.912 ms, 1.945 ms] 650.06 µs (50.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.276 ms [1.263 ms, 1.288 ms] -
iast 3.327 ms [3.28 ms, 3.374 ms] 2.051 ms (160.8%)
iast_FULL 6.516 ms [6.447 ms, 6.584 ms] 5.24 ms (410.7%)
iast_GLOBAL 3.631 ms [3.571 ms, 3.691 ms] 2.355 ms (184.6%)
profiling 2.348 ms [2.324 ms, 2.373 ms] 1.073 ms (84.1%)
tracing 1.935 ms [1.918 ms, 1.953 ms] 659.403 µs (51.7%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.013 ms) : 17834, 18193
.   : milestone, 18013,
appsec (18.655 ms) : 18467, 18843
.   : milestone, 18655,
code_origins (18.848 ms) : 18659, 19038
.   : milestone, 18848,
iast (17.888 ms) : 17715, 18062
.   : milestone, 17888,
profiling (18.281 ms) : 18097, 18464
.   : milestone, 18281,
tracing (17.929 ms) : 17755, 18104
.   : milestone, 17929,
section candidate
no_agent (19.146 ms) : 18949, 19342
.   : milestone, 19146,
appsec (18.559 ms) : 18373, 18745
.   : milestone, 18559,
code_origins (18.047 ms) : 17867, 18227
.   : milestone, 18047,
iast (18.709 ms) : 18522, 18896
.   : milestone, 18709,
profiling (20.165 ms) : 19964, 20365
.   : milestone, 20165,
tracing (17.753 ms) : 17577, 17928
.   : milestone, 17753,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.013 ms [17.834 ms, 18.193 ms] -
appsec 18.655 ms [18.467 ms, 18.843 ms] 641.887 µs (3.6%)
code_origins 18.848 ms [18.659 ms, 19.038 ms] 834.766 µs (4.6%)
iast 17.888 ms [17.715 ms, 18.062 ms] -125.125 µs (-0.7%)
profiling 18.281 ms [18.097 ms, 18.464 ms] 267.322 µs (1.5%)
tracing 17.929 ms [17.755 ms, 18.104 ms] -83.989 µs (-0.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.146 ms [18.949 ms, 19.342 ms] -
appsec 18.559 ms [18.373 ms, 18.745 ms] -586.929 µs (-3.1%)
code_origins 18.047 ms [17.867 ms, 18.227 ms] -1.099 ms (-5.7%)
iast 18.709 ms [18.522 ms, 18.896 ms] -436.281 µs (-2.3%)
profiling 20.165 ms [19.964 ms, 20.365 ms] 1.019 ms (5.3%)
tracing 17.753 ms [17.577 ms, 17.928 ms] -1.393 ms (-7.3%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master dougqh/list-iteration-benchmark
git_commit_date 1782158081 1782171008
git_commit_sha ea827ba b816fde
release_version 1.64.0-SNAPSHOT~ea827ba0c4 1.61.0-SNAPSHOT~b816fdeed7
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1782173374 1782173374
ci_job_id 1794044755 1794044755
ci_pipeline_id 120371156 120371156
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-exmlhb4b 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-exmlhb4b 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.491 ms) : 1479, 1502
.   : milestone, 1491,
appsec (3.856 ms) : 3631, 4080
.   : milestone, 3856,
iast (2.294 ms) : 2224, 2365
.   : milestone, 2294,
iast_GLOBAL (2.33 ms) : 2258, 2401
.   : milestone, 2330,
profiling (2.563 ms) : 2397, 2728
.   : milestone, 2563,
tracing (2.118 ms) : 2063, 2173
.   : milestone, 2118,
section candidate
no_agent (1.497 ms) : 1486, 1509
.   : milestone, 1497,
appsec (3.815 ms) : 3589, 4040
.   : milestone, 3815,
iast (2.277 ms) : 2208, 2346
.   : milestone, 2277,
iast_GLOBAL (2.318 ms) : 2248, 2387
.   : milestone, 2318,
profiling (2.098 ms) : 2043, 2153
.   : milestone, 2098,
tracing (2.069 ms) : 2016, 2122
.   : milestone, 2069,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.491 ms [1.479 ms, 1.502 ms] -
appsec 3.856 ms [3.631 ms, 4.08 ms] 2.365 ms (158.7%)
iast 2.294 ms [2.224 ms, 2.365 ms] 803.494 µs (53.9%)
iast_GLOBAL 2.33 ms [2.258 ms, 2.401 ms] 839.202 µs (56.3%)
profiling 2.563 ms [2.397 ms, 2.728 ms] 1.072 ms (71.9%)
tracing 2.118 ms [2.063 ms, 2.173 ms] 627.097 µs (42.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.497 ms [1.486 ms, 1.509 ms] -
appsec 3.815 ms [3.589 ms, 4.04 ms] 2.317 ms (154.8%)
iast 2.277 ms [2.208 ms, 2.346 ms] 779.297 µs (52.0%)
iast_GLOBAL 2.318 ms [2.248 ms, 2.387 ms] 820.478 µs (54.8%)
profiling 2.098 ms [2.043 ms, 2.153 ms] 600.682 µs (40.1%)
tracing 2.069 ms [2.016 ms, 2.122 ms] 571.765 µs (38.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.61.0-SNAPSHOT~b816fdeed7, baseline=1.64.0-SNAPSHOT~ea827ba0c4
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.946 s) : 14946000, 14946000
.   : milestone, 14946000,
appsec (14.885 s) : 14885000, 14885000
.   : milestone, 14885000,
iast (18.649 s) : 18649000, 18649000
.   : milestone, 18649000,
iast_GLOBAL (18.034 s) : 18034000, 18034000
.   : milestone, 18034000,
profiling (14.961 s) : 14961000, 14961000
.   : milestone, 14961000,
tracing (14.741 s) : 14741000, 14741000
.   : milestone, 14741000,
section candidate
no_agent (15.423 s) : 15423000, 15423000
.   : milestone, 15423000,
appsec (14.776 s) : 14776000, 14776000
.   : milestone, 14776000,
iast (18.147 s) : 18147000, 18147000
.   : milestone, 18147000,
iast_GLOBAL (18.005 s) : 18005000, 18005000
.   : milestone, 18005000,
profiling (15.464 s) : 15464000, 15464000
.   : milestone, 15464000,
tracing (15.052 s) : 15052000, 15052000
.   : milestone, 15052000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.946 s [14.946 s, 14.946 s] -
appsec 14.885 s [14.885 s, 14.885 s] -61.0 ms (-0.4%)
iast 18.649 s [18.649 s, 18.649 s] 3.703 s (24.8%)
iast_GLOBAL 18.034 s [18.034 s, 18.034 s] 3.088 s (20.7%)
profiling 14.961 s [14.961 s, 14.961 s] 15.0 ms (0.1%)
tracing 14.741 s [14.741 s, 14.741 s] -205.0 ms (-1.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.423 s [15.423 s, 15.423 s] -
appsec 14.776 s [14.776 s, 14.776 s] -647.0 ms (-4.2%)
iast 18.147 s [18.147 s, 18.147 s] 2.724 s (17.7%)
iast_GLOBAL 18.005 s [18.005 s, 18.005 s] 2.582 s (16.7%)
profiling 15.464 s [15.464 s, 15.464 s] 41.0 ms (0.3%)
tracing 15.052 s [15.052 s, 15.052 s] -371.0 ms (-2.4%)

sarahchen6
sarahchen6 reviewed Jun 22, 2026
View reviewed changes
Comment thread internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java Outdated
Comment thread internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java
@datadog-official

This comment has been minimized.

Sign in to view
@dd-octo-sts

dd-octo-sts Bot commented Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.87 s 13.85 s [-0.7%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.91 s 12.96 s [-1.1%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 16.91 s 16.76 s [+0.1%; +1.8%] (maybe worse)
startup:petclinic:iast:Agent 16.92 s 16.87 s [-0.5%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.90 s 16.95 s [-1.4%; +0.7%] (no difference)
startup:petclinic:sca:Agent 16.79 s 16.15 s [-0.5%; +8.5%] (no difference)
startup:petclinic:tracing:Agent 16.04 s 15.94 s [-0.8%; +2.0%] (no difference)

Commit: e67ff0a6 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

sarahchen6
sarahchen6 reviewed Jun 23, 2026
View reviewed changes
Comment thread internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java Outdated
sarahchen6
sarahchen6 reviewed Jun 23, 2026
View reviewed changes
Comment thread internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java Outdated
@Warmup(iterations = 2)
@Measurement(iterations = 3)
@Threads(8)
@State(Scope.Benchmark)

@sarahchen6 sarahchen6 Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be better to use @State(Scope.Thread) here to avoid different benchmarks iterating on the same fields?

@dougqh dougqh Jun 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that might be a more realistic comparison.

For maps, I think the shared map has some merit for the precomputed case, but that doesn't so much apply to lists.

dougqh and others added 2 commits June 23, 2026 13:36
@dougqh @sarahchen6
Update internal-api/src/jmh/java/datadog/trace/util/ListIterationBenc…
600cc9b 
…hmark.java

Co-authored-by: Sarah Chen <[email protected]>
@dougqh @claude
Isolate per-thread collections in ListIterationBenchmark
26c2c0b 
Build each thread's list (and its Elements) in a Scope.Thread @setup so the
manipulate_* mutations stay thread-local. Previously the lists lived in enum
constants shared across all 8 threads, so the benchmark measured cross-thread
contention on Element.num rather than iteration cost. Also bump to @fork(2)
and fix a Javadoc typo.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@dd-octo-sts dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Jun 23, 2026
sarahchen6
sarahchen6 approved these changes Jun 23, 2026
View reviewed changes

@sarahchen6 sarahchen6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume the numbers need to be updated with the new changes, but pre-approving!

@dougqh

dougqh commented Jun 23, 2026

Copy link
Copy Markdown
Contributor Author

I assume the numbers need to be updated with the new changes, but pre-approving!

@sarahchen6
Yes, you assume correctly. I'm going to re-run the benchmark tonight, update the results, and then merge. Thanks for the review.

dougqh and others added 2 commits June 23, 2026 18:35
@dougqh
Merge remote-tracking branch 'origin/master' into dougqh/list-iterati…
a981815 
…on-benchmark
@dougqh @claude
Replace stale results in ListIterationBenchmark with Java 17 numbers
8571e72 
Drop the old (pre-per-thread-state) results table; add a condensed Java 17
block. For ArrayList the direct styles (cstyleFor/forEach/enhanced-for/iterator)
cluster within ~10%; stream() is ~3.6x slower; parallelStream() is catastrophic
for small lists (ForkJoinPool overhead) and erratic.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@dougqh
dougqh added this pull request to the merge queue Jun 24, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-24 14:38:30 UTC ℹ️ Start processing command /merge

2026-06-24 14:38:35 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-06-24 15:40:57 UTC ❌ MergeQueue: This merge request was updated

This PR is rejected because it was updated

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026
@dougqh
dougqh enabled auto-merge June 24, 2026 15:41
@dougqh
dougqh added this pull request to the merge queue Jun 24, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jun 24, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-06-24 16:43:35 UTC ℹ️ Start processing command /merge

2026-06-24 16:43:40 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-06-24 18:05:59 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 24, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 16a5297 into master Jun 24, 2026
583 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the dougqh/list-iteration-benchmark branch June 24, 2026 18:05
@github-actions github-actions Bot added this to the 1.64.0 milestone Jun 24, 2026
TophrC-dd pushed a commit that referenced this pull request Jun 29, 2026
@dougqh @sarahchen6 @TophrC-dd
List iteration benchmark (#10888)
a8648fd 
List iteration benchmark

Merge branch 'master' into dougqh/list-iteration-benchmark

Fixing silly oversight - commented out benchmarks to run one in isolation earlier

A bit of clean-up

Adding missing end ul to doc
Moving iterator benchmarks next to enhancedFor

Merge branch 'master' into dougqh/list-iteration-benchmark

spotless

Merge branch 'dougqh/list-iteration-benchmark' of github.com:DataDog/dd-trace-java into dougqh/list-iteration-benchmark

Update internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java

Co-authored-by: Sarah Chen <[email protected]>

Update internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java

Co-authored-by: Sarah Chen <[email protected]>

Merge branch 'master' into dougqh/list-iteration-benchmark

Update internal-api/src/jmh/java/datadog/trace/util/ListIterationBenchmark.java

Co-authored-by: Sarah Chen <[email protected]>

Isolate per-thread collections in ListIterationBenchmark

Build each thread's list (and its Elements) in a Scope.Thread @setup so the
manipulate_* mutations stay thread-local. Previously the lists lived in enum
constants shared across all 8 threads, so the benchmark measured cross-thread
contention on Element.num rather than iteration cost. Also bump to @fork(2)
and fix a Javadoc typo.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Merge remote-tracking branch 'origin/master' into dougqh/list-iteration-benchmark

Replace stale results in ListIterationBenchmark with Java 17 numbers

Drop the old (pre-per-thread-state) results table; add a condensed Java 17
block. For ArrayList the direct styles (cstyleFor/forEach/enhanced-for/iterator)
cluster within ~10%; stream() is ~3.6x slower; parallelStream() is catastrophic
for small lists (ForkJoinPool overhead) and erratic.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Merge branch 'master' into dougqh/list-iteration-benchmark

Co-authored-by: sarahchen6 <[email protected]>
Co-authored-by: devflow.devflow-routing-intake <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sarahchen6 sarahchen6 sarahchen6 approved these changes
@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: documentation

Projects

None yet

Milestone

1.64.0

Development

Successfully merging this pull request may close these issues.

2 participants

@dougqh @sarahchen6