Fix lettuce tests by randomanderson · Pull Request #8664 · DataDog/dd-trace-java

randomanderson · 2025-04-02T17:01:06Z

Lettuce tests were flawed in a few ways:

They were order dependent. An innocuous change like modifying a constant can cause them all to fail. This is most evident with the shutdown() test which shuts down the server and breaks all tests after it
The lettuce-5 Sync tests were not run at all. The inner classes in Lettuce5SyncClientTest were extending the Async flavor
Much of the setup code was repeated leading to slightly different behavior between tests

This PR fixes all of those issues. The setup code has been moved into a base class and the server is restarted between tests to prevent order dependence.

Motivation

I stumbled upon the broken tests while migrating to Gitlab. Gitlab has a different test order so it always broke.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

# Conflicts: # dd-java-agent/instrumentation/lettuce-5/src/test/groovy/Lettuce5AsyncClientTest.groovy # dd-java-agent/instrumentation/lettuce-5/src/test/groovy/Lettuce5ReactiveClientTest.groovy # dd-java-agent/instrumentation/lettuce-5/src/test/groovy/Lettuce5SyncClientTest.groovy

github-actions · 2025-04-02T17:01:19Z

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

randomanderson · 2025-04-02T17:02:20Z

+            "$Tags.PEER_HOSTNAME" redisServer.getHost()
+            "$Tags.PEER_PORT" port
            "$Tags.DB_TYPE" "redis"
+            "db.redis.dbIndex" 0


These tests were never run so the fact that these tags weren't verified was hidden

pr-commenter · 2025-04-02T17:48:21Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	landerson/fix-lettuce-tests
git_commit_date	1743610492	1743617949
git_commit_sha	`09b7c65`	`305b97e`
release_version	1.48.0-SNAPSHOT~09b7c65d37	1.48.0-SNAPSHOT~305b97e86b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1743620934	1743620934
ci_job_id	877933541	877933541
ci_pipeline_id	60851859	60851859
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-pjtrxvqe-project-304-concurrent-0-p5sgp7im 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-pjtrxvqe-project-304-concurrent-0-p5sgp7im 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 66 metrics, 5 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058037
Total [baseline] (10.519 s) : 0, 10519056
Agent [candidate] (1.065 s) : 0, 1064856
Total [candidate] (10.456 s) : 0, 10456214
section appsec
Agent [baseline] (1.197 s) : 0, 1196568
Total [baseline] (10.764 s) : 0, 10763736
Agent [candidate] (1.203 s) : 0, 1202696
Total [candidate] (10.904 s) : 0, 10903921
section iast
Agent [baseline] (1.188 s) : 0, 1188444
Total [baseline] (11.1 s) : 0, 11099600
Agent [candidate] (1.181 s) : 0, 1181009
Total [candidate] (11.018 s) : 0, 11018307
section profiling
Agent [baseline] (1.276 s) : 0, 1276099
Total [baseline] (10.926 s) : 0, 10926445
Agent [candidate] (1.279 s) : 0, 1279170
Total [candidate] (10.951 s) : 0, 10951069

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.197 s	138.531 ms (13.1%)
Agent	iast	1.188 s	130.407 ms (12.3%)
Agent	profiling	1.276 s	218.062 ms (20.6%)
Total	tracing	10.519 s	-
Total	appsec	10.764 s	244.68 ms (2.3%)
Total	iast	11.1 s	580.544 ms (5.5%)
Total	profiling	10.926 s	407.389 ms (3.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	appsec	1.203 s	137.841 ms (12.9%)
Agent	iast	1.181 s	116.153 ms (10.9%)
Agent	profiling	1.279 s	214.314 ms (20.1%)
Total	tracing	10.456 s	-
Total	appsec	10.904 s	447.707 ms (4.3%)
Total	iast	11.018 s	562.093 ms (5.4%)
Total	profiling	10.951 s	494.856 ms (4.7%)

gantt
    title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (723.833 ms) : 0, 723833
BytebuddyAgent [candidate] (727.958 ms) : 0, 727958
GlobalTracer [baseline] (241.283 ms) : 0, 241283
GlobalTracer [candidate] (242.055 ms) : 0, 242055
AppSec [baseline] (54.778 ms) : 0, 54778
AppSec [candidate] (55.369 ms) : 0, 55369
Debugger [baseline] (4.428 ms) : 0, 4428
Debugger [candidate] (4.456 ms) : 0, 4456
Remote Config [baseline] (706.47 µs) : 0, 706
Remote Config [candidate] (710.741 µs) : 0, 711
Telemetry [baseline] (12.388 ms) : 0, 12388
Telemetry [candidate] (13.573 ms) : 0, 13573
section appsec
BytebuddyAgent [baseline] (739.79 ms) : 0, 739790
BytebuddyAgent [candidate] (743.039 ms) : 0, 743039
GlobalTracer [baseline] (236.459 ms) : 0, 236459
GlobalTracer [candidate] (238.007 ms) : 0, 238007
IAST [baseline] (21.541 ms) : 0, 21541
IAST [candidate] (22.141 ms) : 0, 22141
AppSec [baseline] (175.993 ms) : 0, 175993
AppSec [candidate] (176.757 ms) : 0, 176757
Debugger [baseline] (4.261 ms) : 0, 4261
Debugger [candidate] (4.301 ms) : 0, 4301
Remote Config [baseline] (633.365 µs) : 0, 633
Remote Config [candidate] (650.163 µs) : 0, 650
Telemetry [baseline] (8.235 ms) : 0, 8235
Telemetry [candidate] (8.284 ms) : 0, 8284
section iast
BytebuddyAgent [baseline] (844.624 ms) : 0, 844624
BytebuddyAgent [candidate] (839.003 ms) : 0, 839003
GlobalTracer [baseline] (230.717 ms) : 0, 230717
GlobalTracer [candidate] (229.715 ms) : 0, 229715
IAST [baseline] (24.405 ms) : 0, 24405
IAST [candidate] (22.717 ms) : 0, 22717
AppSec [baseline] (54.58 ms) : 0, 54580
AppSec [candidate] (55.84 ms) : 0, 55840
Debugger [baseline] (4.139 ms) : 0, 4139
Debugger [candidate] (4.075 ms) : 0, 4075
Remote Config [baseline] (607.654 µs) : 0, 608
Remote Config [candidate] (597.205 µs) : 0, 597
Telemetry [baseline] (8.793 ms) : 0, 8793
Telemetry [candidate] (8.652 ms) : 0, 8652
section profiling
BytebuddyAgent [baseline] (711.326 ms) : 0, 711326
BytebuddyAgent [candidate] (712.793 ms) : 0, 712793
GlobalTracer [baseline] (348.884 ms) : 0, 348884
GlobalTracer [candidate] (350.117 ms) : 0, 350117
AppSec [baseline] (54.145 ms) : 0, 54145
AppSec [candidate] (53.201 ms) : 0, 53201
Debugger [baseline] (4.293 ms) : 0, 4293
Debugger [candidate] (4.303 ms) : 0, 4303
Remote Config [baseline] (687.793 µs) : 0, 688
Remote Config [candidate] (699.868 µs) : 0, 700
Telemetry [baseline] (8.948 ms) : 0, 8948
Telemetry [candidate] (8.961 ms) : 0, 8961
ProfilingAgent [baseline] (102.167 ms) : 0, 102167
ProfilingAgent [candidate] (103.337 ms) : 0, 103337
Profiling [baseline] (102.192 ms) : 0, 102192
Profiling [candidate] (103.364 ms) : 0, 103364

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.051 s) : 0, 1050862
Total [baseline] (8.67 s) : 0, 8669515
Agent [candidate] (1.059 s) : 0, 1058822
Total [candidate] (8.71 s) : 0, 8709992
section iast
Agent [baseline] (1.18 s) : 0, 1180403
Total [baseline] (9.287 s) : 0, 9287352
Agent [candidate] (1.197 s) : 0, 1197309
Total [candidate] (9.301 s) : 0, 9300656
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1179797
Total [baseline] (9.212 s) : 0, 9211536
Agent [candidate] (1.19 s) : 0, 1189817
Total [candidate] (9.263 s) : 0, 9263496
section iast_TELEMETRY_OFF
Agent [baseline] (1.185 s) : 0, 1185397
Total [baseline] (9.307 s) : 0, 9307191
Agent [candidate] (1.188 s) : 0, 1187959
Total [candidate] (9.316 s) : 0, 9316032

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.051 s	-
Agent	iast	1.18 s	129.541 ms (12.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.18 s	128.935 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.185 s	134.536 ms (12.8%)
Total	tracing	8.67 s	-
Total	iast	9.287 s	617.837 ms (7.1%)
Total	iast_HARDCODED_SECRET_DISABLED	9.212 s	542.021 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.307 s	637.675 ms (7.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.059 s	-
Agent	iast	1.197 s	138.487 ms (13.1%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.19 s	130.995 ms (12.4%)
Agent	iast_TELEMETRY_OFF	1.188 s	129.137 ms (12.2%)
Total	tracing	8.71 s	-
Total	iast	9.301 s	590.664 ms (6.8%)
Total	iast_HARDCODED_SECRET_DISABLED	9.263 s	553.503 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.316 s	606.04 ms (7.0%)

gantt
    title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (719.26 ms) : 0, 719260
BytebuddyAgent [candidate] (720.905 ms) : 0, 720905
GlobalTracer [baseline] (239.505 ms) : 0, 239505
GlobalTracer [candidate] (241.631 ms) : 0, 241631
AppSec [baseline] (54.385 ms) : 0, 54385
AppSec [candidate] (55.542 ms) : 0, 55542
Debugger [baseline] (4.392 ms) : 0, 4392
Debugger [candidate] (4.415 ms) : 0, 4415
Remote Config [baseline] (696.755 µs) : 0, 697
Remote Config [candidate] (715.722 µs) : 0, 716
Telemetry [baseline] (12.177 ms) : 0, 12177
Telemetry [candidate] (15.131 ms) : 0, 15131
section iast
BytebuddyAgent [baseline] (838.226 ms) : 0, 838226
BytebuddyAgent [candidate] (851.83 ms) : 0, 851830
GlobalTracer [baseline] (229.536 ms) : 0, 229536
GlobalTracer [candidate] (231.721 ms) : 0, 231721
AppSec [baseline] (55.929 ms) : 0, 55929
AppSec [candidate] (56.378 ms) : 0, 56378
Debugger [baseline] (4.162 ms) : 0, 4162
Debugger [candidate] (4.196 ms) : 0, 4196
Remote Config [baseline] (609.883 µs) : 0, 610
Remote Config [candidate] (607.29 µs) : 0, 607
Telemetry [baseline] (8.785 ms) : 0, 8785
Telemetry [candidate] (8.838 ms) : 0, 8838
IAST [baseline] (22.728 ms) : 0, 22728
IAST [candidate] (23.119 ms) : 0, 23119
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (837.648 ms) : 0, 837648
BytebuddyAgent [candidate] (845.076 ms) : 0, 845076
GlobalTracer [baseline] (229.897 ms) : 0, 229897
GlobalTracer [candidate] (231.173 ms) : 0, 231173
AppSec [baseline] (55.632 ms) : 0, 55632
AppSec [candidate] (56.119 ms) : 0, 56119
Debugger [baseline] (4.15 ms) : 0, 4150
Debugger [candidate] (4.179 ms) : 0, 4179
Remote Config [baseline] (614.89 µs) : 0, 615
Remote Config [candidate] (601.096 µs) : 0, 601
Telemetry [baseline] (8.739 ms) : 0, 8739
Telemetry [candidate] (8.843 ms) : 0, 8843
IAST [baseline] (22.674 ms) : 0, 22674
IAST [candidate] (23.203 ms) : 0, 23203
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (841.525 ms) : 0, 841525
BytebuddyAgent [candidate] (843.768 ms) : 0, 843768
GlobalTracer [baseline] (230.997 ms) : 0, 230997
GlobalTracer [candidate] (231.383 ms) : 0, 231383
AppSec [baseline] (56.397 ms) : 0, 56397
AppSec [candidate] (56.129 ms) : 0, 56129
Debugger [baseline] (4.183 ms) : 0, 4183
Debugger [candidate] (4.21 ms) : 0, 4210
Remote Config [baseline] (597.859 µs) : 0, 598
Remote Config [candidate] (599.714 µs) : 0, 600
Telemetry [baseline] (8.648 ms) : 0, 8648
Telemetry [candidate] (8.679 ms) : 0, 8679
IAST [baseline] (22.503 ms) : 0, 22503
IAST [candidate] (22.623 ms) : 0, 22623

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-04-02T18:38:59	2025-04-02T18:46:43
git_branch	master	landerson/fix-lettuce-tests
git_commit_date	1743610492	1743617949
git_commit_sha	`09b7c65`	`305b97e`
release_version	1.48.0-SNAPSHOT~09b7c65d37	1.48.0-SNAPSHOT~305b97e86b
start_time	2025-04-02T18:38:46	2025-04-02T18:46:29

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1743620000	1743620000
ci_job_id	877933542	877933542
ci_pipeline_id	60851859	60851859
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-pjtrxvqe-project-304-concurrent-1-dnzcgmpy 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-pjtrxvqe-project-304-concurrent-1-dnzcgmpy 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 14 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.36 ms) : 1340, 1381
.   : milestone, 1360,
appsec (1.746 ms) : 1722, 1769
.   : milestone, 1746,
appsec_no_iast (1.738 ms) : 1716, 1761
.   : milestone, 1738,
code_origins (1.65 ms) : 1623, 1678
.   : milestone, 1650,
iast (1.506 ms) : 1481, 1531
.   : milestone, 1506,
profiling (1.514 ms) : 1491, 1537
.   : milestone, 1514,
tracing (1.51 ms) : 1486, 1534
.   : milestone, 1510,
section candidate
no_agent (1.358 ms) : 1340, 1377
.   : milestone, 1358,
appsec (1.731 ms) : 1708, 1755
.   : milestone, 1731,
appsec_no_iast (1.717 ms) : 1694, 1741
.   : milestone, 1717,
code_origins (1.693 ms) : 1667, 1720
.   : milestone, 1693,
iast (1.511 ms) : 1486, 1537
.   : milestone, 1511,
profiling (1.522 ms) : 1498, 1546
.   : milestone, 1522,
tracing (1.495 ms) : 1470, 1520
.   : milestone, 1495,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.36 ms [1.34 ms, 1.381 ms]	-
appsec	1.746 ms [1.722 ms, 1.769 ms]	385.271 µs (28.3%)
appsec_no_iast	1.738 ms [1.716 ms, 1.761 ms]	377.906 µs (27.8%)
code_origins	1.65 ms [1.623 ms, 1.678 ms]	289.918 µs (21.3%)
iast	1.506 ms [1.481 ms, 1.531 ms]	145.609 µs (10.7%)
profiling	1.514 ms [1.491 ms, 1.537 ms]	153.593 µs (11.3%)
tracing	1.51 ms [1.486 ms, 1.534 ms]	149.775 µs (11.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.358 ms [1.34 ms, 1.377 ms]	-
appsec	1.731 ms [1.708 ms, 1.755 ms]	373.165 µs (27.5%)
appsec_no_iast	1.717 ms [1.694 ms, 1.741 ms]	359.036 µs (26.4%)
code_origins	1.693 ms [1.667 ms, 1.72 ms]	335.169 µs (24.7%)
iast	1.511 ms [1.486 ms, 1.537 ms]	153.14 µs (11.3%)
profiling	1.522 ms [1.498 ms, 1.546 ms]	163.766 µs (12.1%)
tracing	1.495 ms [1.47 ms, 1.52 ms]	136.859 µs (10.1%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37
    dateFormat X
    axisFormat %s
section baseline
no_agent (373.5 µs) : 354, 393
.   : milestone, 373,
iast (508.165 µs) : 486, 530
.   : milestone, 508,
iast_FULL (733.821 µs) : 712, 756
.   : milestone, 734,
iast_GLOBAL (563.457 µs) : 540, 587
.   : milestone, 563,
iast_HARDCODED_SECRET_DISABLED (507.132 µs) : 486, 529
.   : milestone, 507,
iast_INACTIVE (457.386 µs) : 436, 478
.   : milestone, 457,
iast_TELEMETRY_OFF (504.079 µs) : 482, 526
.   : milestone, 504,
tracing (455.247 µs) : 435, 476
.   : milestone, 455,
section candidate
no_agent (381.079 µs) : 360, 402
.   : milestone, 381,
iast (513.864 µs) : 492, 536
.   : milestone, 514,
iast_FULL (728.771 µs) : 707, 751
.   : milestone, 729,
iast_GLOBAL (561.917 µs) : 539, 584
.   : milestone, 562,
iast_HARDCODED_SECRET_DISABLED (514.917 µs) : 493, 537
.   : milestone, 515,
iast_INACTIVE (461.359 µs) : 440, 483
.   : milestone, 461,
iast_TELEMETRY_OFF (506.061 µs) : 484, 529
.   : milestone, 506,
tracing (456.513 µs) : 436, 477
.   : milestone, 457,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	373.5 µs [353.58 µs, 393.42 µs]	-
iast	508.165 µs [486.431 µs, 529.9 µs]	134.666 µs (36.1%)
iast_FULL	733.821 µs [711.72 µs, 755.921 µs]	360.321 µs (96.5%)
iast_GLOBAL	563.457 µs [540.179 µs, 586.735 µs]	189.957 µs (50.9%)
iast_HARDCODED_SECRET_DISABLED	507.132 µs [485.585 µs, 528.678 µs]	133.632 µs (35.8%)
iast_INACTIVE	457.386 µs [436.355 µs, 478.417 µs]	83.886 µs (22.5%)
iast_TELEMETRY_OFF	504.079 µs [482.079 µs, 526.079 µs]	130.579 µs (35.0%)
tracing	455.247 µs [434.677 µs, 475.817 µs]	81.747 µs (21.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	381.079 µs [360.306 µs, 401.851 µs]	-
iast	513.864 µs [491.541 µs, 536.186 µs]	132.785 µs (34.8%)
iast_FULL	728.771 µs [706.733 µs, 750.808 µs]	347.692 µs (91.2%)
iast_GLOBAL	561.917 µs [539.386 µs, 584.447 µs]	180.838 µs (47.5%)
iast_HARDCODED_SECRET_DISABLED	514.917 µs [492.713 µs, 537.121 µs]	133.838 µs (35.1%)
iast_INACTIVE	461.359 µs [440.011 µs, 482.707 µs]	80.281 µs (21.1%)
iast_TELEMETRY_OFF	506.061 µs [483.536 µs, 528.586 µs]	124.982 µs (32.8%)
tracing	456.513 µs [435.936 µs, 477.09 µs]	75.434 µs (19.8%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	landerson/fix-lettuce-tests
git_commit_date	1743610492	1743617949
git_commit_sha	`09b7c65`	`305b97e`
release_version	1.48.0-SNAPSHOT~09b7c65d37	1.48.0-SNAPSHOT~305b97e86b

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1743620648	1743620648
ci_job_id	877933543	877933543
ci_pipeline_id	60851859	60851859
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-x-j3wh1g-project-304-concurrent-0-syifxhxf 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-x-j3wh1g-project-304-concurrent-0-syifxhxf 6.8.0-1024-aws #26~22.04.1-Ubuntu SMP Wed Feb 19 06:54:57 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.484 ms) : 1472, 1495
.   : milestone, 1484,
appsec (2.385 ms) : 2339, 2432
.   : milestone, 2385,
iast (2.177 ms) : 2118, 2236
.   : milestone, 2177,
iast_GLOBAL (2.213 ms) : 2154, 2272
.   : milestone, 2213,
profiling (2.013 ms) : 1966, 2060
.   : milestone, 2013,
tracing (1.992 ms) : 1947, 2038
.   : milestone, 1992,
section candidate
no_agent (1.486 ms) : 1475, 1498
.   : milestone, 1486,
appsec (2.401 ms) : 2353, 2449
.   : milestone, 2401,
iast (2.163 ms) : 2104, 2222
.   : milestone, 2163,
iast_GLOBAL (2.217 ms) : 2157, 2277
.   : milestone, 2217,
profiling (2.034 ms) : 1985, 2083
.   : milestone, 2034,
tracing (2.037 ms) : 1989, 2086
.   : milestone, 2037,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.484 ms [1.472 ms, 1.495 ms]	-
appsec	2.385 ms [2.339 ms, 2.432 ms]	901.903 µs (60.8%)
iast	2.177 ms [2.118 ms, 2.236 ms]	693.118 µs (46.7%)
iast_GLOBAL	2.213 ms [2.154 ms, 2.272 ms]	729.45 µs (49.2%)
profiling	2.013 ms [1.966 ms, 2.06 ms]	529.626 µs (35.7%)
tracing	1.992 ms [1.947 ms, 2.038 ms]	508.66 µs (34.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.486 ms [1.475 ms, 1.498 ms]	-
appsec	2.401 ms [2.353 ms, 2.449 ms]	914.847 µs (61.6%)
iast	2.163 ms [2.104 ms, 2.222 ms]	677.019 µs (45.6%)
iast_GLOBAL	2.217 ms [2.157 ms, 2.277 ms]	731.228 µs (49.2%)
profiling	2.034 ms [1.985 ms, 2.083 ms]	547.983 µs (36.9%)
tracing	2.037 ms [1.989 ms, 2.086 ms]	551.322 µs (37.1%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~305b97e86b, baseline=1.48.0-SNAPSHOT~09b7c65d37
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.853 s) : 14853000, 14853000
.   : milestone, 14853000,
appsec (15.148 s) : 15148000, 15148000
.   : milestone, 15148000,
iast (18.702 s) : 18702000, 18702000
.   : milestone, 18702000,
iast_GLOBAL (17.712 s) : 17712000, 17712000
.   : milestone, 17712000,
profiling (14.943 s) : 14943000, 14943000
.   : milestone, 14943000,
tracing (14.685 s) : 14685000, 14685000
.   : milestone, 14685000,
section candidate
no_agent (15.422 s) : 15422000, 15422000
.   : milestone, 15422000,
appsec (14.727 s) : 14727000, 14727000
.   : milestone, 14727000,
iast (18.859 s) : 18859000, 18859000
.   : milestone, 18859000,
iast_GLOBAL (18.066 s) : 18066000, 18066000
.   : milestone, 18066000,
profiling (15.19 s) : 15190000, 15190000
.   : milestone, 15190000,
tracing (15.137 s) : 15137000, 15137000
.   : milestone, 15137000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.853 s [14.853 s, 14.853 s]	-
appsec	15.148 s [15.148 s, 15.148 s]	295.0 ms (2.0%)
iast	18.702 s [18.702 s, 18.702 s]	3.849 s (25.9%)
iast_GLOBAL	17.712 s [17.712 s, 17.712 s]	2.859 s (19.2%)
profiling	14.943 s [14.943 s, 14.943 s]	90.0 ms (0.6%)
tracing	14.685 s [14.685 s, 14.685 s]	-168.0 ms (-1.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.422 s [15.422 s, 15.422 s]	-
appsec	14.727 s [14.727 s, 14.727 s]	-695.0 ms (-4.5%)
iast	18.859 s [18.859 s, 18.859 s]	3.437 s (22.3%)
iast_GLOBAL	18.066 s [18.066 s, 18.066 s]	2.644 s (17.1%)
profiling	15.19 s [15.19 s, 15.19 s]	-232.0 ms (-1.5%)
tracing	15.137 s [15.137 s, 15.137 s]	-285.0 ms (-1.8%)

amarziali

Thanks

randomanderson added 3 commits April 2, 2025 12:47

cleanup imports

d7592b8

create asyncCommands

7dad598

randomanderson requested a review from a team as a code owner April 2, 2025 17:01

randomanderson commented Apr 2, 2025

View reviewed changes

codenarc

305b97e

randomanderson requested a review from amarziali April 2, 2025 19:04

mcculls approved these changes Apr 2, 2025

View reviewed changes

mcculls added tag: no release notes Changes to exclude from release notes type: refactoring inst: lettuce Lettuce instrumentation labels Apr 2, 2025

amarziali approved these changes Apr 3, 2025

View reviewed changes

randomanderson added the comp: testing Testing label Apr 3, 2025

randomanderson merged commit 6d284ce into master Apr 3, 2025
253 of 255 checks passed

randomanderson deleted the landerson/fix-lettuce-tests branch April 3, 2025 13:20

github-actions Bot added this to the 1.48.0 milestone Apr 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix lettuce tests#8664

Fix lettuce tests#8664
randomanderson merged 4 commits intomasterfrom
landerson/fix-lettuce-tests

randomanderson commented Apr 2, 2025 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 2, 2025 •

edited

Loading

Uh oh!

randomanderson Apr 2, 2025

Uh oh!

pr-commenter Bot commented Apr 2, 2025 •

edited

Loading

Uh oh!

amarziali left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

randomanderson commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Contributor Checklist

Uh oh!

github-actions Bot commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

randomanderson Apr 2, 2025

Choose a reason for hiding this comment

Uh oh!

pr-commenter Bot commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

amarziali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

randomanderson commented Apr 2, 2025 •

edited

Loading

github-actions Bot commented Apr 2, 2025 •

edited

Loading

pr-commenter Bot commented Apr 2, 2025 •

edited

Loading