Do not switch to agent telemetry client on detecting agent telemetry endpoint availability when intake client is preferred by nikita-tkachenko-datadog · Pull Request #6644 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2024-02-08T11:52:06Z

What Does This Do

Updates telemetry client switch logic.
Prior to the change the logic could switch from intake client to agent client in the following circumstances:

if a telemetry request fails
if agent is available and has telemetry proxy endpoint

With the change the logic is to do the switch if:

if a telemetry request fails
if agent is available, has telemetry proxy endpoint and the tracer is not configured for "agentless" mode.

Motivation

If the tracer is explicitly configured to work in agentless mode, the intake telemetry client should be the preferred one.

Jira ticket: CIVIS-2427

…endpoint availability when intake client is preferred

pr-commenter · 2024-02-08T12:33:51Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/telemetry-use-intake-client-by-default
git_commit_date	1707478894	1707484565
git_commit_sha	`80420af`	`6913074`
release_version	1.31.0-SNAPSHOT~80420afa01	1.30.0-SNAPSHOT~6913074b37

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1707487563	1707487563
ci_job_id	430403630	430403630
ci_pipeline_id	28028680	28028680
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.30.0-SNAPSHOT~6913074b37, baseline=1.31.0-SNAPSHOT~80420afa01

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1057864
Total [baseline] (9.405 s) : 0, 9405376
Agent [candidate] (1.068 s) : 0, 1068087
Total [candidate] (9.423 s) : 0, 9422926
section appsec
Agent [baseline] (1.156 s) : 0, 1155669
Total [baseline] (9.463 s) : 0, 9463419
Agent [candidate] (1.156 s) : 0, 1155973
Total [candidate] (9.459 s) : 0, 9458606
section iast
Agent [baseline] (1.192 s) : 0, 1191679
Total [baseline] (9.72 s) : 0, 9720457
Agent [candidate] (1.185 s) : 0, 1185059
Total [candidate] (9.79 s) : 0, 9790379
section profiling
Agent [baseline] (1.291 s) : 0, 1291090
Total [baseline] (9.59 s) : 0, 9589551
Agent [candidate] (1.284 s) : 0, 1283539
Total [candidate] (9.585 s) : 0, 9584674

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.058 s	-
Agent	appsec	1.156 s	97.805 ms (9.2%)
Agent	iast	1.192 s	133.815 ms (12.6%)
Agent	profiling	1.291 s	233.226 ms (22.0%)
Total	tracing	9.405 s	-
Total	appsec	9.463 s	58.043 ms (0.6%)
Total	iast	9.72 s	315.081 ms (3.4%)
Total	profiling	9.59 s	184.175 ms (2.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.068 s	-
Agent	appsec	1.156 s	87.886 ms (8.2%)
Agent	iast	1.185 s	116.972 ms (11.0%)
Agent	profiling	1.284 s	215.453 ms (20.2%)
Total	tracing	9.423 s	-
Total	appsec	9.459 s	35.68 ms (0.4%)
Total	iast	9.79 s	367.453 ms (3.9%)
Total	profiling	9.585 s	161.748 ms (1.7%)

gantt
    title petclinic - break down per module: candidate=1.30.0-SNAPSHOT~6913074b37, baseline=1.31.0-SNAPSHOT~80420afa01

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (666.196 ms) : 0, 666196
BytebuddyAgent [candidate] (672.874 ms) : 0, 672874
GlobalTracer [baseline] (297.711 ms) : 0, 297711
GlobalTracer [candidate] (300.259 ms) : 0, 300259
AppSec [baseline] (51.466 ms) : 0, 51466
AppSec [candidate] (52.04 ms) : 0, 52040
Remote Config [baseline] (692.92 µs) : 0, 693
Remote Config [candidate] (714.829 µs) : 0, 715
Telemetry [baseline] (7.558 ms) : 0, 7558
Telemetry [candidate] (7.617 ms) : 0, 7617
section appsec
BytebuddyAgent [baseline] (666.759 ms) : 0, 666759
BytebuddyAgent [candidate] (667.281 ms) : 0, 667281
GlobalTracer [baseline] (297.519 ms) : 0, 297519
GlobalTracer [candidate] (297.291 ms) : 0, 297291
AppSec [baseline] (149.612 ms) : 0, 149612
AppSec [candidate] (149.771 ms) : 0, 149771
Remote Config [baseline] (648.868 µs) : 0, 649
Remote Config [candidate] (646.155 µs) : 0, 646
Telemetry [baseline] (6.818 ms) : 0, 6818
Telemetry [candidate] (6.722 ms) : 0, 6722
section iast
BytebuddyAgent [baseline] (784.594 ms) : 0, 784594
BytebuddyAgent [candidate] (778.7 ms) : 0, 778700
GlobalTracer [baseline] (290.594 ms) : 0, 290594
GlobalTracer [candidate] (289.583 ms) : 0, 289583
AppSec [baseline] (50.629 ms) : 0, 50629
AppSec [candidate] (52.123 ms) : 0, 52123
Remote Config [baseline] (634.383 µs) : 0, 634
Remote Config [candidate] (632.238 µs) : 0, 632
Telemetry [baseline] (7.335 ms) : 0, 7335
Telemetry [candidate] (8.132 ms) : 0, 8132
IAST [baseline] (23.301 ms) : 0, 23301
IAST [candidate] (21.485 ms) : 0, 21485
section profiling
BytebuddyAgent [baseline] (672.831 ms) : 0, 672831
BytebuddyAgent [candidate] (669.091 ms) : 0, 669091
GlobalTracer [baseline] (385.471 ms) : 0, 385471
GlobalTracer [candidate] (383.155 ms) : 0, 383155
AppSec [baseline] (52.512 ms) : 0, 52512
AppSec [candidate] (52.412 ms) : 0, 52412
Remote Config [baseline] (669.866 µs) : 0, 670
Remote Config [candidate] (656.818 µs) : 0, 657
Telemetry [baseline] (11.631 ms) : 0, 11631
Telemetry [candidate] (9.979 ms) : 0, 9979
ProfilingAgent [baseline] (112.803 ms) : 0, 112803
ProfilingAgent [candidate] (113.446 ms) : 0, 113446
Profiling [baseline] (112.828 ms) : 0, 112828
Profiling [candidate] (113.471 ms) : 0, 113471

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-02-09T13:42:00	2024-02-09T14:00:59
git_branch	master	nikita-tkachenko/telemetry-use-intake-client-by-default
git_commit_date	1707478894	1707484565
git_commit_sha	`80420af`	`6913074`
release_version	1.31.0-SNAPSHOT~80420afa01	1.30.0-SNAPSHOT~6913074b37
start_time	2024-02-09T13:41:47	2024-02-09T14:00:45

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1707487563	1707487563
ci_job_id	430403630	430403630
ci_pipeline_id	28028680	28028680
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 16 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.30.0-SNAPSHOT~6913074b37, baseline=1.31.0-SNAPSHOT~80420afa01
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.352 ms) : 1332, 1371
.   : milestone, 1352,
appsec (1.79 ms) : 1764, 1815
.   : milestone, 1790,
iast (1.506 ms) : 1481, 1531
.   : milestone, 1506,
profiling (1.563 ms) : 1537, 1589
.   : milestone, 1563,
tracing (1.481 ms) : 1455, 1506
.   : milestone, 1481,
section candidate
no_agent (1.352 ms) : 1332, 1371
.   : milestone, 1352,
appsec (1.78 ms) : 1755, 1805
.   : milestone, 1780,
iast (1.527 ms) : 1502, 1551
.   : milestone, 1527,
profiling (1.57 ms) : 1544, 1596
.   : milestone, 1570,
tracing (1.504 ms) : 1479, 1529
.   : milestone, 1504,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.352 ms [1.332 ms, 1.371 ms]	-
appsec	1.79 ms [1.764 ms, 1.815 ms]	438.134 µs (32.4%)
iast	1.506 ms [1.481 ms, 1.531 ms]	154.406 µs (11.4%)
profiling	1.563 ms [1.537 ms, 1.589 ms]	211.534 µs (15.7%)
tracing	1.481 ms [1.455 ms, 1.506 ms]	128.941 µs (9.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.352 ms [1.332 ms, 1.371 ms]	-
appsec	1.78 ms [1.755 ms, 1.805 ms]	428.492 µs (31.7%)
iast	1.527 ms [1.502 ms, 1.551 ms]	174.998 µs (12.9%)
profiling	1.57 ms [1.544 ms, 1.596 ms]	218.175 µs (16.1%)
tracing	1.504 ms [1.479 ms, 1.529 ms]	152.563 µs (11.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.30.0-SNAPSHOT~6913074b37, baseline=1.31.0-SNAPSHOT~80420afa01
    dateFormat X
    axisFormat %s
section baseline
no_agent (369.507 µs) : 349, 390
.   : milestone, 370,
iast (486.896 µs) : 465, 508
.   : milestone, 487,
iast_FULL (536.704 µs) : 516, 557
.   : milestone, 537,
iast_GLOBAL (509.929 µs) : 488, 531
.   : milestone, 510,
iast_HARDCODED_SECRET_DISABLED (473.449 µs) : 452, 494
.   : milestone, 473,
iast_INACTIVE (455.032 µs) : 433, 477
.   : milestone, 455,
iast_TELEMETRY_OFF (470.946 µs) : 450, 492
.   : milestone, 471,
tracing (439.404 µs) : 419, 460
.   : milestone, 439,
section candidate
no_agent (366.493 µs) : 347, 386
.   : milestone, 366,
iast (491.995 µs) : 471, 513
.   : milestone, 492,
iast_FULL (547.615 µs) : 527, 569
.   : milestone, 548,
iast_GLOBAL (497.654 µs) : 477, 518
.   : milestone, 498,
iast_HARDCODED_SECRET_DISABLED (473.488 µs) : 453, 494
.   : milestone, 473,
iast_INACTIVE (453.68 µs) : 432, 475
.   : milestone, 454,
iast_TELEMETRY_OFF (478.647 µs) : 457, 500
.   : milestone, 479,
tracing (450.719 µs) : 430, 472
.   : milestone, 451,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	369.507 µs [349.077 µs, 389.938 µs]	-
iast	486.896 µs [465.466 µs, 508.326 µs]	117.389 µs (31.8%)
iast_FULL	536.704 µs [516.232 µs, 557.175 µs]	167.196 µs (45.2%)
iast_GLOBAL	509.929 µs [488.5 µs, 531.359 µs]	140.422 µs (38.0%)
iast_HARDCODED_SECRET_DISABLED	473.449 µs [452.439 µs, 494.459 µs]	103.942 µs (28.1%)
iast_INACTIVE	455.032 µs [433.167 µs, 476.898 µs]	85.525 µs (23.1%)
iast_TELEMETRY_OFF	470.946 µs [450.336 µs, 491.557 µs]	101.439 µs (27.5%)
tracing	439.404 µs [419.211 µs, 459.598 µs]	69.897 µs (18.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.493 µs [346.584 µs, 386.402 µs]	-
iast	491.995 µs [471.286 µs, 512.703 µs]	125.502 µs (34.2%)
iast_FULL	547.615 µs [526.629 µs, 568.602 µs]	181.123 µs (49.4%)
iast_GLOBAL	497.654 µs [476.934 µs, 518.373 µs]	131.161 µs (35.8%)
iast_HARDCODED_SECRET_DISABLED	473.488 µs [453.097 µs, 493.88 µs]	106.996 µs (29.2%)
iast_INACTIVE	453.68 µs [432.065 µs, 475.295 µs]	87.187 µs (23.8%)
iast_TELEMETRY_OFF	478.647 µs [457.391 µs, 499.902 µs]	112.154 µs (30.6%)
tracing	450.719 µs [429.51 µs, 471.928 µs]	84.226 µs (23.0%)

PerfectSlayer

LGTM
Should we have more test about it?

nikita-tkachenko-datadog · 2024-02-09T13:16:56Z

LGTM Should we have more test about it?

Sure, it won't hurt to have a couple of dedicated test cases for it. Added.

Do not switch to agent telemetry client on detecting agent telemetry …

c6864a2

…endpoint availability when intake client is preferred

nikita-tkachenko-datadog added the comp: telemetry Telemetry label Feb 8, 2024

nikita-tkachenko-datadog marked this pull request as ready for review February 8, 2024 12:45

nikita-tkachenko-datadog requested a review from a team as a code owner February 8, 2024 12:45

nikita-tkachenko-datadog requested review from PerfectSlayer and nayeem-kamal February 8, 2024 12:45

PerfectSlayer approved these changes Feb 9, 2024

View reviewed changes

Add some test cases

6913074

nikita-tkachenko-datadog merged commit a3eb733 into master Feb 9, 2024

nikita-tkachenko-datadog deleted the nikita-tkachenko/telemetry-use-intake-client-by-default branch February 9, 2024 14:16

github-actions Bot added this to the 1.30.0 milestone Feb 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not switch to agent telemetry client on detecting agent telemetry endpoint availability when intake client is preferred#6644

Do not switch to agent telemetry client on detecting agent telemetry endpoint availability when intake client is preferred#6644
nikita-tkachenko-datadog merged 2 commits intomasterfrom
nikita-tkachenko/telemetry-use-intake-client-by-default

nikita-tkachenko-datadog commented Feb 8, 2024 •

edited by atlassian Bot

Loading

Uh oh!

pr-commenter Bot commented Feb 8, 2024 •

edited

Loading

Uh oh!

PerfectSlayer left a comment

Uh oh!

nikita-tkachenko-datadog commented Feb 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nikita-tkachenko-datadog commented Feb 8, 2024 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Uh oh!

pr-commenter Bot commented Feb 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

nikita-tkachenko-datadog commented Feb 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikita-tkachenko-datadog commented Feb 8, 2024 •

edited by atlassian Bot

Loading

pr-commenter Bot commented Feb 8, 2024 •

edited

Loading