Send probe statuses to new debugger track by jpbempel · Pull Request #6276 · DataDog/dd-trace-java

jpbempel · 2023-11-24T16:12:57Z

What Does This Do

datadog agent is now exposing a new endpoint: /debugger/v1/diagnostics This endpoint is used to send probe statuses to a dedicated debugger track.
we add detection o this new endpoint and create a dedicated BatchUploader to this endpoint using attachment.
if the endpoint is not available (old datadog agent) we send to the previous url as before.

Motivation

Additional Notes

Jira ticket: DEBUG-1853

pr-commenter · 2023-11-24T16:57:04Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	jpbempel/diagnostic-endpoint
git_commit_date	1702409587	1702412918
git_commit_sha	`1878db6`	`aa39e08`
release_version	1.26.0-SNAPSHOT~1878db6329	1.26.0-SNAPSHOT~aa39e0829b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702416285	1702416285
ci_job_id	388300689	388300689
ci_pipeline_id	24976682	24976682
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 2 performance regressions! Performance is the same for 46 metrics, 6 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:petclinic:profiling:ProfilingAgent	worse [+11.386ms; +21.014ms] or [+8.407%; +15.516%]	151.629ms	135.430ms
scenario:startup:petclinic:profiling:Profiling	worse [+11.377ms; +21.014ms] or [+8.397%; +15.509%]	151.684ms	135.488ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.566 s) : 0, 1566011
Total [baseline] (12.922 s) : 0, 12921758
Agent [candidate] (1.579 s) : 0, 1578921
Total [candidate] (13.055 s) : 0, 13054846
section appsec
Agent [baseline] (1.7 s) : 0, 1699989
Total [baseline] (12.961 s) : 0, 12961215
Agent [candidate] (1.703 s) : 0, 1702520
Total [candidate] (13.035 s) : 0, 13034601
section iast
Agent [baseline] (1.717 s) : 0, 1716950
Total [baseline] (13.275 s) : 0, 13275228
Agent [candidate] (1.72 s) : 0, 1720278
Total [candidate] (13.32 s) : 0, 13319514
section profiling
Agent [baseline] (1.866 s) : 0, 1865796
Total [baseline] (13.311 s) : 0, 13311227
Agent [candidate] (1.889 s) : 0, 1888767
Total [candidate] (13.271 s) : 0, 13271054

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.566 s	-
Agent	appsec	1.7 s	133.978 ms (8.6%)
Agent	iast	1.717 s	150.939 ms (9.6%)
Agent	profiling	1.866 s	299.785 ms (19.1%)
Total	tracing	12.922 s	-
Total	appsec	12.961 s	39.457 ms (0.3%)
Total	iast	13.275 s	353.47 ms (2.7%)
Total	profiling	13.311 s	389.469 ms (3.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.579 s	-
Agent	appsec	1.703 s	123.598 ms (7.8%)
Agent	iast	1.72 s	141.357 ms (9.0%)
Agent	profiling	1.889 s	309.846 ms (19.6%)
Total	tracing	13.055 s	-
Total	appsec	13.035 s	-20.245 ms (-0.2%)
Total	iast	13.32 s	264.668 ms (2.0%)
Total	profiling	13.271 s	216.208 ms (1.7%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (967.955 ms) : 0, 967955
BytebuddyAgent [candidate] (976.598 ms) : 0, 976598
GlobalTracer [baseline] (458.622 ms) : 0, 458622
GlobalTracer [candidate] (462.11 ms) : 0, 462110
AppSec [baseline] (69.069 ms) : 0, 69069
AppSec [candidate] (69.101 ms) : 0, 69101
Remote Config [baseline] (853.3 µs) : 0, 853
Remote Config [candidate] (856.077 µs) : 0, 856
Telemetry [baseline] (9.445 ms) : 0, 9445
Telemetry [candidate] (9.475 ms) : 0, 9475
section appsec
BytebuddyAgent [baseline] (969.9 ms) : 0, 969900
BytebuddyAgent [candidate] (969.836 ms) : 0, 969836
GlobalTracer [baseline] (456.651 ms) : 0, 456651
GlobalTracer [candidate] (458.154 ms) : 0, 458154
AppSec [baseline] (203.496 ms) : 0, 203496
AppSec [candidate] (204.354 ms) : 0, 204354
Remote Config [baseline] (800.216 µs) : 0, 800
Remote Config [candidate] (795.356 µs) : 0, 795
Telemetry [baseline] (9.211 ms) : 0, 9211
Telemetry [candidate] (9.151 ms) : 0, 9151
section iast
BytebuddyAgent [baseline] (1.137 s) : 0, 1136746
BytebuddyAgent [candidate] (1.139 s) : 0, 1138958
GlobalTracer [baseline] (421.061 ms) : 0, 421061
GlobalTracer [candidate] (420.652 ms) : 0, 420652
AppSec [baseline] (66.84 ms) : 0, 66840
AppSec [candidate] (65.792 ms) : 0, 65792
IAST [baseline] (20.64 ms) : 0, 20640
IAST [candidate] (22.063 ms) : 0, 22063
Remote Config [baseline] (765.461 µs) : 0, 765
Remote Config [candidate] (757.422 µs) : 0, 757
Telemetry [baseline] (10.855 ms) : 0, 10855
Telemetry [candidate] (11.536 ms) : 0, 11536
section profiling
BytebuddyAgent [baseline] (988.857 ms) : 0, 988857
BytebuddyAgent [candidate] (995.469 ms) : 0, 995469
GlobalTracer [baseline] (571.61 ms) : 0, 571610
GlobalTracer [candidate] (570.941 ms) : 0, 570941
AppSec [baseline] (69.514 ms) : 0, 69514
AppSec [candidate] (69.677 ms) : 0, 69677
Remote Config [baseline] (849.358 µs) : 0, 849
Remote Config [candidate] (853.499 µs) : 0, 853
Telemetry [baseline] (9.767 ms) : 0, 9767
Telemetry [candidate] (9.818 ms) : 0, 9818
ProfilingAgent [baseline] (135.43 ms) : 0, 135430
ProfilingAgent [candidate] (151.629 ms) : 0, 151629
Profiling [baseline] (135.488 ms) : 0, 135488
Profiling [candidate] (151.684 ms) : 0, 151684

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.568 s) : 0, 1567880
Total [baseline] (12.248 s) : 0, 12247685
Agent [candidate] (1.576 s) : 0, 1575807
Total [candidate] (12.26 s) : 0, 12259664
section iast
Agent [baseline] (1.725 s) : 0, 1725337
Total [baseline] (12.99 s) : 0, 12989836
Agent [candidate] (1.721 s) : 0, 1720749
Total [candidate] (12.934 s) : 0, 12934314
section iast_TELEMETRY_OFF
Agent [baseline] (1.706 s) : 0, 1706497
Total [baseline] (12.893 s) : 0, 12892725
Agent [candidate] (1.708 s) : 0, 1707998
Total [candidate] (12.912 s) : 0, 12912191

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.568 s	-
Agent	iast	1.725 s	157.457 ms (10.0%)
Agent	iast_TELEMETRY_OFF	1.706 s	138.616 ms (8.8%)
Total	tracing	12.248 s	-
Total	iast	12.99 s	742.152 ms (6.1%)
Total	iast_TELEMETRY_OFF	12.893 s	645.04 ms (5.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.576 s	-
Agent	iast	1.721 s	144.943 ms (9.2%)
Agent	iast_TELEMETRY_OFF	1.708 s	132.191 ms (8.4%)
Total	tracing	12.26 s	-
Total	iast	12.934 s	674.651 ms (5.5%)
Total	iast_TELEMETRY_OFF	12.912 s	652.527 ms (5.3%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (969.645 ms) : 0, 969645
BytebuddyAgent [candidate] (974.925 ms) : 0, 974925
GlobalTracer [baseline] (458.39 ms) : 0, 458390
GlobalTracer [candidate] (460.771 ms) : 0, 460771
AppSec [baseline] (69.263 ms) : 0, 69263
AppSec [candidate] (69.269 ms) : 0, 69269
Remote Config [baseline] (852.81 µs) : 0, 853
Remote Config [candidate] (858.428 µs) : 0, 858
Telemetry [baseline] (9.396 ms) : 0, 9396
Telemetry [candidate] (9.402 ms) : 0, 9402
section iast
BytebuddyAgent [baseline] (1.141 s) : 0, 1141490
BytebuddyAgent [candidate] (1.138 s) : 0, 1138347
GlobalTracer [baseline] (422.393 ms) : 0, 422393
GlobalTracer [candidate] (420.827 ms) : 0, 420827
AppSec [baseline] (66.306 ms) : 0, 66306
AppSec [candidate] (66.728 ms) : 0, 66728
IAST [baseline] (24.62 ms) : 0, 24620
IAST [candidate] (25.215 ms) : 0, 25215
Remote Config [baseline] (777.459 µs) : 0, 777
Remote Config [candidate] (761.673 µs) : 0, 762
Telemetry [baseline] (9.304 ms) : 0, 9304
Telemetry [candidate] (8.468 ms) : 0, 8468
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (1.125 s) : 0, 1124735
BytebuddyAgent [candidate] (1.127 s) : 0, 1126921
GlobalTracer [baseline] (420.661 ms) : 0, 420661
GlobalTracer [candidate] (421.132 ms) : 0, 421132
AppSec [baseline] (66.648 ms) : 0, 66648
AppSec [candidate] (66.724 ms) : 0, 66724
IAST [baseline] (22.64 ms) : 0, 22640
IAST [candidate] (21.081 ms) : 0, 21081
Remote Config [baseline] (762.255 µs) : 0, 762
Remote Config [candidate] (751.155 µs) : 0, 751
Telemetry [baseline] (10.668 ms) : 0, 10668
Telemetry [candidate] (10.802 ms) : 0, 10802

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-12T20:56:31	2023-12-12T21:19:11
git_branch	master	jpbempel/diagnostic-endpoint
git_commit_date	1702409587	1702412918
git_commit_sha	`1878db6`	`aa39e08`
release_version	1.26.0-SNAPSHOT~1878db6329	1.26.0-SNAPSHOT~aa39e0829b
start_time	2023-12-12T20:56:14	2023-12-12T21:18:55

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702416285	1702416285
ci_job_id	388300689	388300689
ci_pipeline_id	24976682	24976682
cpu_model	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 5 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.738 ms) : 1714, 1763
.   : milestone, 1738,
appsec (2.228 ms) : 2195, 2260
.   : milestone, 2228,
iast (1.946 ms) : 1913, 1980
.   : milestone, 1946,
profiling (1.909 ms) : 1875, 1942
.   : milestone, 1909,
tracing (1.91 ms) : 1877, 1944
.   : milestone, 1910,
section candidate
no_agent (1.724 ms) : 1699, 1749
.   : milestone, 1724,
appsec (2.217 ms) : 2184, 2250
.   : milestone, 2217,
iast (1.935 ms) : 1902, 1968
.   : milestone, 1935,
profiling (1.947 ms) : 1909, 1985
.   : milestone, 1947,
tracing (1.896 ms) : 1862, 1930
.   : milestone, 1896,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.738 ms [1.714 ms, 1.763 ms]	-
appsec	2.228 ms [2.195 ms, 2.26 ms]	489.29 µs (28.1%)
iast	1.946 ms [1.913 ms, 1.98 ms]	208.082 µs (12.0%)
profiling	1.909 ms [1.875 ms, 1.942 ms]	170.47 µs (9.8%)
tracing	1.91 ms [1.877 ms, 1.944 ms]	172.189 µs (9.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.724 ms [1.699 ms, 1.749 ms]	-
appsec	2.217 ms [2.184 ms, 2.25 ms]	493.034 µs (28.6%)
iast	1.935 ms [1.902 ms, 1.968 ms]	210.885 µs (12.2%)
profiling	1.947 ms [1.909 ms, 1.985 ms]	222.904 µs (12.9%)
tracing	1.896 ms [1.862 ms, 1.93 ms]	172.167 µs (10.0%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~aa39e0829b, baseline=1.26.0-SNAPSHOT~1878db6329
    dateFormat X
    axisFormat %s
section baseline
no_agent (449.899 µs) : 421, 479
.   : milestone, 450,
iast (600.686 µs) : 570, 631
.   : milestone, 601,
iast_FULL (663.692 µs) : 633, 695
.   : milestone, 664,
iast_INACTIVE (538.95 µs) : 509, 569
.   : milestone, 539,
iast_TELEMETRY_OFF (568.319 µs) : 537, 600
.   : milestone, 568,
tracing (531.063 µs) : 501, 562
.   : milestone, 531,
section candidate
no_agent (450.796 µs) : 421, 480
.   : milestone, 451,
iast (575.073 µs) : 544, 606
.   : milestone, 575,
iast_FULL (660.042 µs) : 628, 692
.   : milestone, 660,
iast_INACTIVE (541.77 µs) : 512, 572
.   : milestone, 542,
iast_TELEMETRY_OFF (567.099 µs) : 536, 598
.   : milestone, 567,
tracing (544.172 µs) : 513, 575
.   : milestone, 544,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	449.899 µs [420.741 µs, 479.058 µs]	-
iast	600.686 µs [570.373 µs, 631.0 µs]	150.787 µs (33.5%)
iast_FULL	663.692 µs [632.679 µs, 694.706 µs]	213.793 µs (47.5%)
iast_INACTIVE	538.95 µs [508.791 µs, 569.109 µs]	89.05 µs (19.8%)
iast_TELEMETRY_OFF	568.319 µs [536.816 µs, 599.821 µs]	118.419 µs (26.3%)
tracing	531.063 µs [500.541 µs, 561.585 µs]	81.163 µs (18.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	450.796 µs [421.137 µs, 480.455 µs]	-
iast	575.073 µs [543.876 µs, 606.27 µs]	124.277 µs (27.6%)
iast_FULL	660.042 µs [628.472 µs, 691.611 µs]	209.246 µs (46.4%)
iast_INACTIVE	541.77 µs [511.527 µs, 572.014 µs]	90.974 µs (20.2%)
iast_TELEMETRY_OFF	567.099 µs [536.254 µs, 597.944 µs]	116.303 µs (25.8%)
tracing	544.172 µs [513.018 µs, 575.326 µs]	93.376 µs (20.7%)

cimi · 2023-11-28T14:13:12Z

The DEBUGGER track is only available for internal orgs at the moment, we need to wait to merge this PR until we GA the track to all orgs.

cimi

Thank you, looks good! 🙌 Left some comments, nothing blocking.

The debugger track has been enabled for customer orgs but we're still in standby; we can start sending traffic soon.

Let's test this build in our backend services next week and if everything is fine we can merge.

cimi · 2023-12-07T11:49:22Z

Nit: it would be nicer if the feature discovery API returned an optional path with the endpoint, so it's guaranteed that what we return matches what we checked for.

Since this is not specific to our endpoint, I assume it requires a lot of refactoring - I'm not suggesting to do this in this PR 🙂

cimi · 2023-12-07T12:00:55Z

Nit: we could keep the old constructor and default to the old endpoint inside it - that way we don't need to update everywhere.

Don't feel strongly about it, if you want to keep this explicit that's fine.

want to keep it explicit, easier also for tests

cimi · 2023-12-07T12:07:34Z

Nit: could we replace this check with ddAgentFeaturesDiscovery.supportsDebuggerDiagnostics() in the caller of the constructor of this class?

Instead of matching the endpoint against a string, just ask the caller to say if this is should be a multipart upload or not.

cimi · 2023-12-07T12:15:25Z

Nit: we could make an overload for this method with no tags and avoid passing the empty string.

in fact I need to pass them like for the non-multipart one

cimi · 2023-12-07T12:15:42Z

I don't understand this change - we're doing the tags sanitization twice now, do we still need the serviceName field?

is service sanitization done on the backend?
for serviceName, .NET is doing the same. ping @ojung

I think the sanitization is mostly used in places where we put things into the tags. There is backend-specific sanitization in the backend, but I think generally what we should do is be consistent. So if we're sanitizing the service name in other places, we should sanitize it in the same way here to be able to reliably link back to a service from any tags/places that a service name is uploaded. Does that make sense?

every time we use the config.getServiceName in the tracer we sanitize it:

https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/ConfigurationUpdater.java#L78

https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/DebuggerProductChangesListener.java#L81

https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/ProbeStatus.java#L246

datadog agent is now exposing a new endpoint: /debugger/v1/diagnostics This endpoint is used to send probe statuses to a dedicated debugger track. we add detection o this new endpoint and create a dedicated BatchUploader to this endpoint using attachment. if the endpoint is not available (old datadog agent) we send to the previous url as before.

jpbempel requested review from a team as code owners November 24, 2023 16:12

jpbempel requested review from am312, mcculls and shatzi and removed request for a team November 24, 2023 16:12

jpbempel added comp: debugger Dynamic Instrumentation type: feature Enhancements and improvements labels Nov 24, 2023

jpbempel requested a review from cimi November 24, 2023 16:13

cimi approved these changes Dec 7, 2023

View reviewed changes

jpbempel force-pushed the jpbempel/diagnostic-endpoint branch 2 times, most recently from e4a327c to 8275795 Compare December 12, 2023 09:52

jpbempel added 3 commits December 12, 2023 21:28

remove event from BatchUploader

f2dc394

pass multipart usage as constructor param

aa39e08

jpbempel force-pushed the jpbempel/diagnostic-endpoint branch from 8275795 to aa39e08 Compare December 12, 2023 20:28

jpbempel merged commit b306862 into master Dec 14, 2023

jpbempel deleted the jpbempel/diagnostic-endpoint branch December 14, 2023 13:29

github-actions Bot added this to the 1.26.0 milestone Dec 14, 2023

Uh oh!

Conversation

jpbempel commented Nov 24, 2023 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Nov 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

cimi commented Nov 28, 2023

Uh oh!

cimi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jpbempel Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jpbempel commented Nov 24, 2023 •

edited by atlassian Bot

Loading

pr-commenter Bot commented Nov 24, 2023 •

edited

Loading

jpbempel Dec 11, 2023 •

edited

Loading