Send only snapshots to Debugger Track #9501

jpbempel · 2025-09-10T07:00:25Z

What Does This Do

We introduce a separate BatchUploader instance to send capureSnapshot=true snapshots (including variables) to dedicated datadog endpoint for DEBUGGER track.
SnapshotSink now expect 2 BatchUploaders (lowRate for snapshots and highRate for templated logs only)
BatchUploader now take a name to identify in debug logs which instance is used.
Smoke tests are updated to use processRequest everywhere and get rid of retrieveSnapshots that is not correct.

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: DEBUG-4444

pr-commenter · 2025-09-10T07:17:33Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1757574243	1757574588
end_time	2025-09-11T07:05:25	2025-09-11T07:11:09
git_branch	master	jpbempel/split-snapshots-logs
git_commit_sha	`036fe91`	`0d9742a`
start_time	2025-09-11T07:04:04	2025-09-11T07:09:49

See matching parameters

	Baseline	Candidate
ci_job_id	1123893790	1123893790
ci_pipeline_id	76155160	76155160
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1757573642	1757573642

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-35.563µs; +41.860µs] or [-12.388%; +14.582%]	unstable [-48.769µs; +59.725µs] or [-14.809%; +18.136%]	unstable [-59.483µs; +67.966µs] or [-17.242%; +19.701%]	unstable [-190.738µs; +271.181µs] or [-19.935%; +28.342%]	same
scenario:basic	same	same	same	unstable [-159.296µs; +24.782µs] or [-20.349%; +3.166%]	unstable [-186.925op/s; +186.925op/s] or [-7.103%; +7.103%]
scenario:loop	same	same	unsure [+0.498µs; +11.589µs] or [+0.006%; +0.129%]	same	same

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (329.318 µs) : 285, 374
.   : milestone, 329,
basic (280.906 µs) : 274, 288
.   : milestone, 281,
loop (8.96 ms) : 8955, 8965
.   : milestone, 8960,
section candidate
noprobe (334.796 µs) : 277, 393
.   : milestone, 335,
basic (279.657 µs) : 273, 286
.   : milestone, 280,
loop (8.964 ms) : 8959, 8969
.   : milestone, 8964,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	329.318 µs [285.067 µs, 373.569 µs]
basic	280.906 µs [274.014 µs, 287.799 µs]
loop	8.96 ms [8.955 ms, 8.965 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	334.796 µs [276.707 µs, 392.885 µs]
basic	279.657 µs [273.105 µs, 286.209 µs]
loop	8.964 ms [8.959 ms, 8.969 ms]

datadog-official · 2025-09-10T07:19:57Z

🎯 Code Coverage
• Patch Coverage: 68.89%
• Total Coverage: 62.75% (+2.62%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 0d9742a | Docs | Was this helpful? Give us feedback!}

pr-commenter · 2025-09-10T07:32:20Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/split-snapshots-logs
git_commit_date	1757519723	1757573642
git_commit_sha	`036fe91`	`0d9742a`
release_version	1.54.0-SNAPSHOT~036fe910ec	1.54.0-SNAPSHOT~0d9742aa73

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1757575496	1757575496
ci_job_id	1123893783	1123893783
ci_pipeline_id	76155160	76155160
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-fijfotn8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-fijfotn8 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058406
Total [baseline] (10.806 s) : 0, 10806497
Agent [candidate] (1.046 s) : 0, 1045899
Total [candidate] (10.662 s) : 0, 10662470
section appsec
Agent [baseline] (1.232 s) : 0, 1232011
Total [baseline] (10.907 s) : 0, 10907026
Agent [candidate] (1.228 s) : 0, 1227718
Total [candidate] (10.863 s) : 0, 10862754
section iast
Agent [baseline] (1.189 s) : 0, 1188675
Total [baseline] (11.005 s) : 0, 11005266
Agent [candidate] (1.192 s) : 0, 1191513
Total [candidate] (10.951 s) : 0, 10951134
section profiling
Agent [baseline] (1.201 s) : 0, 1201183
Total [baseline] (10.977 s) : 0, 10977047
Agent [candidate] (1.202 s) : 0, 1202229
Total [candidate] (10.931 s) : 0, 10930833

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.232 s	173.605 ms (16.4%)
Agent	iast	1.189 s	130.269 ms (12.3%)
Agent	profiling	1.201 s	142.777 ms (13.5%)
Total	tracing	10.806 s	-
Total	appsec	10.907 s	100.53 ms (0.9%)
Total	iast	11.005 s	198.77 ms (1.8%)
Total	profiling	10.977 s	170.55 ms (1.6%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	appsec	1.228 s	181.819 ms (17.4%)
Agent	iast	1.192 s	145.614 ms (13.9%)
Agent	profiling	1.202 s	156.33 ms (14.9%)
Total	tracing	10.662 s	-
Total	appsec	10.863 s	200.284 ms (1.9%)
Total	iast	10.951 s	288.664 ms (2.7%)
Total	profiling	10.931 s	268.362 ms (2.5%)

gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.452 ms) : 0, 1452
crashtracking [candidate] (1.445 ms) : 0, 1445
BytebuddyAgent [baseline] (740.138 ms) : 0, 740138
BytebuddyAgent [candidate] (732.832 ms) : 0, 732832
GlobalTracer [baseline] (245.228 ms) : 0, 245228
GlobalTracer [candidate] (242.805 ms) : 0, 242805
AppSec [baseline] (30.872 ms) : 0, 30872
AppSec [candidate] (30.497 ms) : 0, 30497
Debugger [baseline] (6.515 ms) : 0, 6515
Debugger [candidate] (6.422 ms) : 0, 6422
Remote Config [baseline] (682.327 µs) : 0, 682
Remote Config [candidate] (676.375 µs) : 0, 676
Telemetry [baseline] (12.394 ms) : 0, 12394
Telemetry [candidate] (10.152 ms) : 0, 10152
section appsec
crashtracking [baseline] (1.469 ms) : 0, 1469
crashtracking [candidate] (1.453 ms) : 0, 1453
BytebuddyAgent [baseline] (760.615 ms) : 0, 760615
BytebuddyAgent [candidate] (757.701 ms) : 0, 757701
GlobalTracer [baseline] (237.098 ms) : 0, 237098
GlobalTracer [candidate] (236.394 ms) : 0, 236394
IAST [baseline] (23.836 ms) : 0, 23836
IAST [candidate] (23.676 ms) : 0, 23676
AppSec [baseline] (170.015 ms) : 0, 170015
AppSec [candidate] (170.0 ms) : 0, 170000
Debugger [baseline] (7.669 ms) : 0, 7669
Debugger [candidate] (7.579 ms) : 0, 7579
Remote Config [baseline] (641.602 µs) : 0, 642
Remote Config [candidate] (672.873 µs) : 0, 673
Telemetry [baseline] (9.43 ms) : 0, 9430
Telemetry [candidate] (9.193 ms) : 0, 9193
section iast
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.471 ms) : 0, 1471
BytebuddyAgent [baseline] (857.967 ms) : 0, 857967
BytebuddyAgent [candidate] (860.053 ms) : 0, 860053
GlobalTracer [baseline] (234.298 ms) : 0, 234298
GlobalTracer [candidate] (236.037 ms) : 0, 236037
IAST [baseline] (29.534 ms) : 0, 29534
IAST [candidate] (31.431 ms) : 0, 31431
AppSec [baseline] (28.345 ms) : 0, 28345
AppSec [candidate] (26.361 ms) : 0, 26361
Debugger [baseline] (7.009 ms) : 0, 7009
Debugger [candidate] (6.142 ms) : 0, 6142
Remote Config [baseline] (620.143 µs) : 0, 620
Remote Config [candidate] (608.281 µs) : 0, 608
Telemetry [baseline] (8.301 ms) : 0, 8301
Telemetry [candidate] (8.24 ms) : 0, 8240
section profiling
crashtracking [baseline] (1.437 ms) : 0, 1437
crashtracking [candidate] (1.421 ms) : 0, 1421
BytebuddyAgent [baseline] (762.572 ms) : 0, 762572
BytebuddyAgent [candidate] (763.75 ms) : 0, 763750
GlobalTracer [baseline] (223.397 ms) : 0, 223397
GlobalTracer [candidate] (223.937 ms) : 0, 223937
AppSec [baseline] (31.006 ms) : 0, 31006
AppSec [candidate] (30.944 ms) : 0, 30944
Debugger [baseline] (6.634 ms) : 0, 6634
Debugger [candidate] (6.754 ms) : 0, 6754
Remote Config [baseline] (692.11 µs) : 0, 692
Remote Config [candidate] (724.848 µs) : 0, 725
Telemetry [baseline] (16.64 ms) : 0, 16640
Telemetry [candidate] (16.46 ms) : 0, 16460
ProfilingAgent [baseline] (108.089 ms) : 0, 108089
ProfilingAgent [candidate] (107.711 ms) : 0, 107711
Profiling [baseline] (108.822 ms) : 0, 108822
Profiling [candidate] (108.42 ms) : 0, 108420

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064408
Total [baseline] (8.692 s) : 0, 8692125
Agent [candidate] (1.063 s) : 0, 1062573
Total [candidate] (8.665 s) : 0, 8665449
section iast
Agent [baseline] (1.181 s) : 0, 1181417
Total [baseline] (9.312 s) : 0, 9312370
Agent [candidate] (1.183 s) : 0, 1183389
Total [candidate] (9.382 s) : 0, 9382316

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	iast	1.181 s	117.008 ms (11.0%)
Total	tracing	8.692 s	-
Total	iast	9.312 s	620.246 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.063 s	-
Agent	iast	1.183 s	120.816 ms (11.4%)
Total	tracing	8.665 s	-
Total	iast	9.382 s	716.867 ms (8.3%)

gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.479 ms) : 0, 1479
crashtracking [candidate] (1.474 ms) : 0, 1474
BytebuddyAgent [baseline] (744.142 ms) : 0, 744142
BytebuddyAgent [candidate] (742.435 ms) : 0, 742435
GlobalTracer [baseline] (245.194 ms) : 0, 245194
GlobalTracer [candidate] (245.293 ms) : 0, 245293
AppSec [baseline] (31.11 ms) : 0, 31110
AppSec [candidate] (30.899 ms) : 0, 30899
Debugger [baseline] (6.518 ms) : 0, 6518
Debugger [candidate] (6.503 ms) : 0, 6503
Remote Config [baseline] (692.77 µs) : 0, 693
Remote Config [candidate] (685.036 µs) : 0, 685
Telemetry [baseline] (13.947 ms) : 0, 13947
Telemetry [candidate] (14.021 ms) : 0, 14021
section iast
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.459 ms) : 0, 1459
BytebuddyAgent [baseline] (852.508 ms) : 0, 852508
BytebuddyAgent [candidate] (854.116 ms) : 0, 854116
GlobalTracer [baseline] (234.932 ms) : 0, 234932
GlobalTracer [candidate] (233.567 ms) : 0, 233567
AppSec [baseline] (27.909 ms) : 0, 27909
AppSec [candidate] (29.463 ms) : 0, 29463
Debugger [baseline] (6.075 ms) : 0, 6075
Debugger [candidate] (6.136 ms) : 0, 6136
Remote Config [baseline] (600.744 µs) : 0, 601
Remote Config [candidate] (598.895 µs) : 0, 599
Telemetry [baseline] (8.286 ms) : 0, 8286
Telemetry [candidate] (8.354 ms) : 0, 8354
IAST [baseline] (28.682 ms) : 0, 28682
IAST [candidate] (28.661 ms) : 0, 28661

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/split-snapshots-logs
git_commit_date	1757519723	1757573642
git_commit_sha	`036fe91`	`0d9742a`
release_version	1.54.0-SNAPSHOT~036fe910ec	1.54.0-SNAPSHOT~0d9742aa73

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1757575145	1757575145
ci_job_id	1123893784	1123893784
ci_pipeline_id	76155160	76155160
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-dc5yifx2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-dc5yifx2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	worse [+297.566µs; +609.753µs] or [+3.428%; +7.025%]	unstable [-91.693op/s; +39.568op/s] or [-17.159%; +7.405%]	9.133ms	508.312op/s	8.679ms	534.375op/s
scenario:load:petclinic:profiling:high_load	better [-2.139ms; -1.140ms] or [-4.338%; -2.312%]	unstable [-4.059op/s; +10.584op/s] or [-4.274%; +11.145%]	47.665ms	98.225op/s	49.305ms	94.963op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.218 ms) : 4170, 4266
.   : milestone, 4218,
iast (9.511 ms) : 9355, 9668
.   : milestone, 9511,
iast_FULL (14.119 ms) : 13841, 14397
.   : milestone, 14119,
iast_GLOBAL (10.753 ms) : 10552, 10955
.   : milestone, 10753,
profiling (8.679 ms) : 8542, 8817
.   : milestone, 8679,
tracing (7.802 ms) : 7682, 7921
.   : milestone, 7802,
section candidate
no_agent (4.265 ms) : 4214, 4316
.   : milestone, 4265,
iast (9.438 ms) : 9283, 9594
.   : milestone, 9438,
iast_FULL (13.707 ms) : 13435, 13978
.   : milestone, 13707,
iast_GLOBAL (10.538 ms) : 10349, 10727
.   : milestone, 10538,
profiling (9.133 ms) : 8981, 9285
.   : milestone, 9133,
tracing (7.777 ms) : 7664, 7890
.   : milestone, 7777,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.218 ms [4.17 ms, 4.266 ms]	-
iast	9.511 ms [9.355 ms, 9.668 ms]	5.294 ms (125.5%)
iast_FULL	14.119 ms [13.841 ms, 14.397 ms]	9.901 ms (234.7%)
iast_GLOBAL	10.753 ms [10.552 ms, 10.955 ms]	6.536 ms (155.0%)
profiling	8.679 ms [8.542 ms, 8.817 ms]	4.462 ms (105.8%)
tracing	7.802 ms [7.682 ms, 7.921 ms]	3.584 ms (85.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.265 ms [4.214 ms, 4.316 ms]	-
iast	9.438 ms [9.283 ms, 9.594 ms]	5.173 ms (121.3%)
iast_FULL	13.707 ms [13.435 ms, 13.978 ms]	9.442 ms (221.4%)
iast_GLOBAL	10.538 ms [10.349 ms, 10.727 ms]	6.273 ms (147.1%)
profiling	9.133 ms [8.981 ms, 9.285 ms]	4.868 ms (114.1%)
tracing	7.777 ms [7.664 ms, 7.89 ms]	3.512 ms (82.3%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.919 ms) : 36614, 37224
.   : milestone, 36919,
appsec (49.509 ms) : 49055, 49963
.   : milestone, 49509,
code_origins (45.143 ms) : 44756, 45530
.   : milestone, 45143,
iast (45.494 ms) : 45089, 45899
.   : milestone, 45494,
profiling (49.305 ms) : 48801, 49808
.   : milestone, 49305,
tracing (42.748 ms) : 42402, 43095
.   : milestone, 42748,
section candidate
no_agent (36.758 ms) : 36456, 37060
.   : milestone, 36758,
appsec (48.578 ms) : 48162, 48993
.   : milestone, 48578,
code_origins (46.409 ms) : 46001, 46816
.   : milestone, 46409,
iast (45.109 ms) : 44716, 45501
.   : milestone, 45109,
profiling (47.665 ms) : 47244, 48087
.   : milestone, 47665,
tracing (43.915 ms) : 43551, 44280
.   : milestone, 43915,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.919 ms [36.614 ms, 37.224 ms]	-
appsec	49.509 ms [49.055 ms, 49.963 ms]	12.59 ms (34.1%)
code_origins	45.143 ms [44.756 ms, 45.53 ms]	8.224 ms (22.3%)
iast	45.494 ms [45.089 ms, 45.899 ms]	8.575 ms (23.2%)
profiling	49.305 ms [48.801 ms, 49.808 ms]	12.386 ms (33.5%)
tracing	42.748 ms [42.402 ms, 43.095 ms]	5.829 ms (15.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.758 ms [36.456 ms, 37.06 ms]	-
appsec	48.578 ms [48.162 ms, 48.993 ms]	11.819 ms (32.2%)
code_origins	46.409 ms [46.001 ms, 46.816 ms]	9.65 ms (26.3%)
iast	45.109 ms [44.716 ms, 45.501 ms]	8.351 ms (22.7%)
profiling	47.665 ms [47.244 ms, 48.087 ms]	10.907 ms (29.7%)
tracing	43.915 ms [43.551 ms, 44.28 ms]	7.157 ms (19.5%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/split-snapshots-logs
git_commit_date	1757519723	1757573642
git_commit_sha	`036fe91`	`0d9742a`
release_version	1.54.0-SNAPSHOT~036fe910ec	1.54.0-SNAPSHOT~0d9742aa73

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1757575663	1757575663
ci_job_id	1123893785	1123893785
ci_pipeline_id	76155160	76155160
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-6wsggqq0 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-6wsggqq0 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 0 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:dacapo:tomcat:appsec	better [-1.388ms; -1.047ms] or [-37.400%; -28.213%]	2.495ms	3.712ms

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.619 s) : 15619000, 15619000
.   : milestone, 15619000,
appsec (15.021 s) : 15021000, 15021000
.   : milestone, 15021000,
iast (18.205 s) : 18205000, 18205000
.   : milestone, 18205000,
iast_GLOBAL (18.03 s) : 18030000, 18030000
.   : milestone, 18030000,
profiling (15.763 s) : 15763000, 15763000
.   : milestone, 15763000,
tracing (14.923 s) : 14923000, 14923000
.   : milestone, 14923000,
section candidate
no_agent (15.02 s) : 15020000, 15020000
.   : milestone, 15020000,
appsec (15.034 s) : 15034000, 15034000
.   : milestone, 15034000,
iast (18.36 s) : 18360000, 18360000
.   : milestone, 18360000,
iast_GLOBAL (17.941 s) : 17941000, 17941000
.   : milestone, 17941000,
profiling (15.719 s) : 15719000, 15719000
.   : milestone, 15719000,
tracing (14.956 s) : 14956000, 14956000
.   : milestone, 14956000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.619 s [15.619 s, 15.619 s]	-
appsec	15.021 s [15.021 s, 15.021 s]	-598.0 ms (-3.8%)
iast	18.205 s [18.205 s, 18.205 s]	2.586 s (16.6%)
iast_GLOBAL	18.03 s [18.03 s, 18.03 s]	2.411 s (15.4%)
profiling	15.763 s [15.763 s, 15.763 s]	144.0 ms (0.9%)
tracing	14.923 s [14.923 s, 14.923 s]	-696.0 ms (-4.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.02 s [15.02 s, 15.02 s]	-
appsec	15.034 s [15.034 s, 15.034 s]	14.0 ms (0.1%)
iast	18.36 s [18.36 s, 18.36 s]	3.34 s (22.2%)
iast_GLOBAL	17.941 s [17.941 s, 17.941 s]	2.921 s (19.4%)
profiling	15.719 s [15.719 s, 15.719 s]	699.0 ms (4.7%)
tracing	14.956 s [14.956 s, 14.956 s]	-64.0 ms (-0.4%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~0d9742aa73, baseline=1.54.0-SNAPSHOT~036fe910ec
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1458, 1481
.   : milestone, 1470,
appsec (3.712 ms) : 3495, 3930
.   : milestone, 3712,
iast (2.198 ms) : 2135, 2261
.   : milestone, 2198,
iast_GLOBAL (2.238 ms) : 2175, 2302
.   : milestone, 2238,
profiling (2.055 ms) : 2003, 2107
.   : milestone, 2055,
tracing (2.004 ms) : 1956, 2053
.   : milestone, 2004,
section candidate
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.495 ms) : 2442, 2547
.   : milestone, 2495,
iast (2.181 ms) : 2118, 2243
.   : milestone, 2181,
iast_GLOBAL (2.237 ms) : 2174, 2300
.   : milestone, 2237,
profiling (2.041 ms) : 1990, 2091
.   : milestone, 2041,
tracing (2.003 ms) : 1955, 2052
.   : milestone, 2003,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.458 ms, 1.481 ms]	-
appsec	3.712 ms [3.495 ms, 3.93 ms]	2.243 ms (152.6%)
iast	2.198 ms [2.135 ms, 2.261 ms]	728.023 µs (49.5%)
iast_GLOBAL	2.238 ms [2.175 ms, 2.302 ms]	768.515 µs (52.3%)
profiling	2.055 ms [2.003 ms, 2.107 ms]	585.486 µs (39.8%)
tracing	2.004 ms [1.956 ms, 2.053 ms]	534.826 µs (36.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.482 ms]	-
appsec	2.495 ms [2.442 ms, 2.547 ms]	1.024 ms (69.7%)
iast	2.181 ms [2.118 ms, 2.243 ms]	710.76 µs (48.3%)
iast_GLOBAL	2.237 ms [2.174 ms, 2.3 ms]	766.88 µs (52.2%)
profiling	2.041 ms [1.99 ms, 2.091 ms]	570.488 µs (38.8%)
tracing	2.003 ms [1.955 ms, 2.052 ms]	533.292 µs (36.3%)

AlexeyKuznetsov-DD · 2025-09-10T14:57:28Z

dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/DebuggerTransformer.java

+                    SnapshotSink.RETRY_POLICY),
+                new BatchUploader(
+                    "Logs",
+                    config,
+                    config.getFinalDebuggerSnapshotUrl(),
+                    SnapshotSink.RETRY_POLICY)),


I can see that we are using RETRY_POLICY always? Can it be some sort of default?
Also I have a feeling that RETRY_POLICY should be in BatchUploader class - WDYT?

for SymDB the policy is different see

dd-trace-java/dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/sink/SymbolSink.java

Line 32 in cb7783b

public static final BatchUploader.RetryPolicy RETRY_POLICY = new BatchUploader.RetryPolicy(10);

We introduce a separate BatchUploader instance to send capureSnapshot=true snapshots (including variables) to dedicated datadog endpoint for DEBUGGER track. SnapshotSink now expect 2 BatchUploaders (lowRate for snapshots and highRate for templated logs only) BatchUploader now take a name to identify in debug logs which instance is used. Smoke tests are updated to use processRequest everywhere and get rid of retrieveSnapshots that is not correct.

jpbempel added the type: enhancement Enhancements and improvements label Sep 10, 2025

jpbempel requested review from a team as code owners September 10, 2025 07:00

jpbempel added the comp: debugger Dynamic Instrumentation label Sep 10, 2025

jpbempel requested review from evanchooly and mcculls and removed request for a team September 10, 2025 07:00

jpbempel force-pushed the jpbempel/split-snapshots-logs branch from f289477 to cc0281a Compare September 10, 2025 08:13

amarziali approved these changes Sep 10, 2025

View reviewed changes

AlexeyKuznetsov-DD reviewed Sep 10, 2025

View reviewed changes

shatzi approved these changes Sep 10, 2025

View reviewed changes

jpbempel force-pushed the jpbempel/split-snapshots-logs branch from cc0281a to ab0fec5 Compare September 11, 2025 05:39

jpbempel force-pushed the jpbempel/split-snapshots-logs branch from ab0fec5 to 0d9742a Compare September 11, 2025 06:54

jpbempel merged commit 0007547 into master Sep 11, 2025
510 checks passed

jpbempel deleted the jpbempel/split-snapshots-logs branch September 11, 2025 11:54

github-actions bot added this to the 1.54.0 milestone Sep 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Send only snapshots to Debugger Track #9501

Send only snapshots to Debugger Track #9501

Uh oh!

jpbempel commented Sep 10, 2025 •

edited by atlassian bot

Loading

Uh oh!

pr-commenter bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

datadog-official bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Sep 10, 2025 •

edited

Loading

Uh oh!

AlexeyKuznetsov-DD Sep 10, 2025

Uh oh!

jpbempel Sep 10, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Send only snapshots to Debugger Track #9501

Send only snapshots to Debugger Track #9501

Uh oh!

Conversation

jpbempel commented Sep 10, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

datadog-official bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

AlexeyKuznetsov-DD Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

jpbempel Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jpbempel commented Sep 10, 2025 •

edited by atlassian bot

Loading

pr-commenter bot commented Sep 10, 2025 •

edited

Loading

datadog-official bot commented Sep 10, 2025 •

edited

Loading

pr-commenter bot commented Sep 10, 2025 •

edited

Loading