Skip to content

Report dd-trace-java and its dependencies to telemetry#5698

Merged
smola merged 3 commits into
masterfrom
smola/report-tracer-deps
Aug 17, 2023
Merged

Report dd-trace-java and its dependencies to telemetry#5698
smola merged 3 commits into
masterfrom
smola/report-tracer-deps

Conversation

@smola

@smola smola commented Aug 9, 2023 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Member

What Does This Do

  • Remove the telemetry check we previously added to avoid reporting dd-trace-java vendored dependencies to telemetry.
  • Amend the exclusion of META-INF/maven to not include pom.properties (introduced in 5346d91), which we use to report dependencies to telemetry.

Motivation

  • In the early days of the telemetry subsystem for dd-trace-java, we excluded dd-trace-java's own dependencies from scanning, since they were not part of the customer application, and their presence could be confusing. Effectively, this meant we did not report them within Vulnerability Management, unless they were also used by the customer application.
  • Later fixes to our build system, unrelated to this issue, removed pom.properties from dd-trace-java jar. pom.properties are not needed to run the tracer, but they hold metadata used by most vulnerability scanners to detect dependencies. This made dd-trace-java dependencies invisible to many vulnerability scanners since dd-trace-java v1.0.0.
  • We decided to undo both changes, so that both DataDog's and any 3rd party vulnerability scanner preserve the ability to accurately detect dd-trace-java dependencies, and their vulnerabilities.
  • IMPORTANT: This means some scanners will start reporting vulnerabilities in dd-trace-java v1.20.0, even if these dependencies were already present.

Additional Notes

  • This leads to a ~0.5ms regression in telemetry startup, presumably because now loading classes from the tracer hits the regular codepath that hits DDCache#computeIfAbsent for seenDomains.
  • APPSEC-10763

@smola smola added the comp: telemetry Telemetry label Aug 9, 2023
@smola
smola requested a review from a team as a code owner August 9, 2023 07:53
@smola
smola requested review from a team and ValentinZakharov August 9, 2023 07:53
@smola
smola force-pushed the smola/report-tracer-deps branch 2 times, most recently from de530a7 to 1f06b9d Compare August 9, 2023 15:40
@smola
smola requested review from a team and DDJavierSantos and removed request for a team and ValentinZakharov August 10, 2023 07:21
@smola
smola force-pushed the smola/report-tracer-deps branch from 1f06b9d to 1acd1d7 Compare August 10, 2023 07:25
@pr-commenter

pr-commenter Bot commented Aug 10, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~9dd556a58b 1.20.0-SNAPSHOT~71a6ad3822
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 8 performance regressions! Performance is the same for 54 cases.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:insecure-bank:appsec:Telemetry worse
[+486.473µs; +669.569µs] or [+9.416%; +12.959%]		 5.745ms 5.167ms
scenario:insecure-bank:iast:Telemetry worse
[+420.096µs; +669.280µs] or [+8.132%; +12.956%]		 5.710ms 5.166ms
scenario:insecure-bank:profiling:Telemetry worse
[+447.368µs; +659.108µs] or [+8.300%; +12.229%]		 5.943ms 5.390ms
scenario:insecure-bank:tracing:Telemetry worse
[+400.760µs; +608.592µs] or [+7.321%; +11.117%]		 5.979ms 5.474ms
scenario:petclinic:appsec:Telemetry worse
[+399.294µs; +665.863µs] or [+7.678%; +12.804%]		 5.733ms 5.201ms
scenario:petclinic:iast:Telemetry worse
[+408.902µs; +695.284µs] or [+7.879%; +13.397%]		 5.742ms 5.190ms
scenario:petclinic:profiling:Telemetry worse
[+434.364µs; +744.176µs] or [+8.147%; +13.957%]		 5.921ms 5.332ms
scenario:petclinic:tracing:Telemetry worse
[+461.125µs; +644.405µs] or [+8.482%; +11.854%]		 5.989ms 5.436ms
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (987.23 ms) : 0, 987230
Total [baseline] (9.193 s) : 0, 9193318
Agent [candidate] (984.084 ms) : 0, 984084
Total [candidate] (9.176 s) : 0, 9176016
section appsec
Agent [baseline] (1.077 s) : 0, 1077454
Total [baseline] (9.32 s) : 0, 9320107
Agent [candidate] (1.075 s) : 0, 1075404
Total [candidate] (9.282 s) : 0, 9281832
section iast
Agent [baseline] (1.085 s) : 0, 1085058
Total [baseline] (9.442 s) : 0, 9441790
Agent [candidate] (1.087 s) : 0, 1087170
Total [candidate] (9.454 s) : 0, 9453946
section profiling
Agent [baseline] (1.169 s) : 0, 1169091
Total [baseline] (9.456 s) : 0, 9455625
Agent [candidate] (1.165 s) : 0, 1164940
Total [candidate] (9.541 s) : 0, 9540710
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 987.23 ms -
Agent appsec 1.077 s 90.224 ms (9.1%)
Agent iast 1.085 s 97.827 ms (9.9%)
Agent profiling 1.169 s 181.861 ms (18.4%)
Total tracing 9.193 s -
Total appsec 9.32 s 126.789 ms (1.4%)
Total iast 9.442 s 248.472 ms (2.7%)
Total profiling 9.456 s 262.306 ms (2.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 984.084 ms -
Agent appsec 1.075 s 91.32 ms (9.3%)
Agent iast 1.087 s 103.085 ms (10.5%)
Agent profiling 1.165 s 180.856 ms (18.4%)
Total tracing 9.176 s -
Total appsec 9.282 s 105.817 ms (1.2%)
Total iast 9.454 s 277.931 ms (3.0%)
Total profiling 9.541 s 364.694 ms (4.0%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (612.461 ms) : 0, 612461
BytebuddyAgent [candidate] (608.848 ms) : 0, 608848
GlobalTracer [baseline] (284.18 ms) : 0, 284180
GlobalTracer [candidate] (283.79 ms) : 0, 283790
AppSec [baseline] (50.175 ms) : 0, 50175
AppSec [candidate] (50.636 ms) : 0, 50636
Remote Config [baseline] (673.437 µs) : 0, 673
Remote Config [candidate] (661.663 µs) : 0, 662
Telemetry [baseline] (5.436 ms) : 0, 5436
Telemetry [candidate] (5.989 ms) : 0, 5989
section appsec
BytebuddyAgent [baseline] (614.721 ms) : 0, 614721
BytebuddyAgent [candidate] (612.964 ms) : 0, 612964
GlobalTracer [baseline] (286.113 ms) : 0, 286113
GlobalTracer [candidate] (285.921 ms) : 0, 285921
AppSec [baseline] (136.26 ms) : 0, 136260
AppSec [candidate] (135.688 ms) : 0, 135688
Remote Config [baseline] (734.117 µs) : 0, 734
Remote Config [candidate] (727.5 µs) : 0, 728
Telemetry [baseline] (5.201 ms) : 0, 5201
Telemetry [candidate] (5.733 ms) : 0, 5733
section iast
BytebuddyAgent [baseline] (713.126 ms) : 0, 713126
BytebuddyAgent [candidate] (714.488 ms) : 0, 714488
GlobalTracer [baseline] (270.987 ms) : 0, 270987
GlobalTracer [candidate] (271.423 ms) : 0, 271423
AppSec [baseline] (46.598 ms) : 0, 46598
AppSec [candidate] (46.409 ms) : 0, 46409
Remote Config [baseline] (605.447 µs) : 0, 605
Remote Config [candidate] (603.081 µs) : 0, 603
Telemetry [baseline] (5.19 ms) : 0, 5190
Telemetry [candidate] (5.742 ms) : 0, 5742
IAST [baseline] (14.263 ms) : 0, 14263
IAST [candidate] (14.188 ms) : 0, 14188
section profiling
ProfilingAgent [baseline] (77.575 ms) : 0, 77575
ProfilingAgent [candidate] (76.681 ms) : 0, 76681
BytebuddyAgent [baseline] (631.282 ms) : 0, 631282
BytebuddyAgent [candidate] (627.557 ms) : 0, 627557
GlobalTracer [baseline] (351.041 ms) : 0, 351041
GlobalTracer [candidate] (351.411 ms) : 0, 351411
AppSec [baseline] (49.557 ms) : 0, 49557
AppSec [candidate] (49.483 ms) : 0, 49483
Remote Config [baseline] (662.264 µs) : 0, 662
Remote Config [candidate] (659.832 µs) : 0, 660
Telemetry [baseline] (5.332 ms) : 0, 5332
Telemetry [candidate] (5.921 ms) : 0, 5921
Profiling [baseline] (77.6 ms) : 0, 77600
Profiling [candidate] (76.705 ms) : 0, 76705
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (992.689 ms) : 0, 992689
Total [baseline] (8.694 s) : 0, 8694162
Agent [candidate] (986.883 ms) : 0, 986883
Total [candidate] (8.641 s) : 0, 8640977
section appsec
Agent [baseline] (1.073 s) : 0, 1073416
Total [baseline] (8.785 s) : 0, 8784758
Agent [candidate] (1.07 s) : 0, 1070298
Total [candidate] (8.752 s) : 0, 8751500
section iast
Agent [baseline] (1.086 s) : 0, 1085695
Total [baseline] (9.164 s) : 0, 9163909
Agent [candidate] (1.086 s) : 0, 1085603
Total [candidate] (9.15 s) : 0, 9149983
section profiling
Agent [baseline] (1.158 s) : 0, 1158213
Total [baseline] (8.873 s) : 0, 8872626
Agent [candidate] (1.16 s) : 0, 1159539
Total [candidate] (8.896 s) : 0, 8896025
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 992.689 ms -
Agent appsec 1.073 s 80.727 ms (8.1%)
Agent iast 1.086 s 93.006 ms (9.4%)
Agent profiling 1.158 s 165.524 ms (16.7%)
Total tracing 8.694 s -
Total appsec 8.785 s 90.596 ms (1.0%)
Total iast 9.164 s 469.747 ms (5.4%)
Total profiling 8.873 s 178.464 ms (2.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 986.883 ms -
Agent appsec 1.07 s 83.415 ms (8.5%)
Agent iast 1.086 s 98.72 ms (10.0%)
Agent profiling 1.16 s 172.656 ms (17.5%)
Total tracing 8.641 s -
Total appsec 8.752 s 110.523 ms (1.3%)
Total iast 9.15 s 509.006 ms (5.9%)
Total profiling 8.896 s 255.048 ms (3.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (615.397 ms) : 0, 615397
BytebuddyAgent [candidate] (610.931 ms) : 0, 610931
GlobalTracer [baseline] (286.522 ms) : 0, 286522
GlobalTracer [candidate] (284.562 ms) : 0, 284562
AppSec [baseline] (50.134 ms) : 0, 50134
AppSec [candidate] (50.362 ms) : 0, 50362
Remote Config [baseline] (666.6 µs) : 0, 667
Remote Config [candidate] (658.76 µs) : 0, 659
Telemetry [baseline] (5.474 ms) : 0, 5474
Telemetry [candidate] (5.979 ms) : 0, 5979
section appsec
BytebuddyAgent [baseline] (614.161 ms) : 0, 614161
BytebuddyAgent [candidate] (610.476 ms) : 0, 610476
GlobalTracer [baseline] (283.931 ms) : 0, 283931
GlobalTracer [candidate] (284.099 ms) : 0, 284099
AppSec [baseline] (134.968 ms) : 0, 134968
AppSec [candidate] (135.133 ms) : 0, 135133
Remote Config [baseline] (723.325 µs) : 0, 723
Remote Config [candidate] (711.084 µs) : 0, 711
Telemetry [baseline] (5.167 ms) : 0, 5167
Telemetry [candidate] (5.745 ms) : 0, 5745
section iast
BytebuddyAgent [baseline] (713.826 ms) : 0, 713826
BytebuddyAgent [candidate] (713.144 ms) : 0, 713144
GlobalTracer [baseline] (271.326 ms) : 0, 271326
GlobalTracer [candidate] (271.172 ms) : 0, 271172
AppSec [baseline] (46.221 ms) : 0, 46221
AppSec [candidate] (46.434 ms) : 0, 46434
Remote Config [baseline] (606.842 µs) : 0, 607
Remote Config [candidate] (607.94 µs) : 0, 608
Telemetry [baseline] (5.166 ms) : 0, 5166
Telemetry [candidate] (5.71 ms) : 0, 5710
IAST [baseline] (14.21 ms) : 0, 14210
IAST [candidate] (14.205 ms) : 0, 14205
section profiling
ProfilingAgent [baseline] (77.018 ms) : 0, 77018
ProfilingAgent [candidate] (75.788 ms) : 0, 75788
BytebuddyAgent [baseline] (623.565 ms) : 0, 623565
BytebuddyAgent [candidate] (624.401 ms) : 0, 624401
GlobalTracer [baseline] (349.318 ms) : 0, 349318
GlobalTracer [candidate] (350.248 ms) : 0, 350248
AppSec [baseline] (49.307 ms) : 0, 49307
AppSec [candidate] (49.619 ms) : 0, 49619
Remote Config [baseline] (666.177 µs) : 0, 666
Remote Config [candidate] (657.72 µs) : 0, 658
Telemetry [baseline] (5.39 ms) : 0, 5390
Telemetry [candidate] (5.943 ms) : 0, 5943
Profiling [baseline] (77.043 ms) : 0, 77043
Profiling [candidate] (75.812 ms) : 0, 75812
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~9dd556a58b 1.20.0-SNAPSHOT~71a6ad3822
config baseline candidate
end_time 2023-08-17T08:33:01 2023-08-17T08:51:09
start_time 2023-08-17T08:32:43 2023-08-17T08:50:52
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.337 ms) : 1317, 1356
.   : milestone, 1337,
appsec (1.648 ms) : 1624, 1673
.   : milestone, 1648,
iast (1.456 ms) : 1433, 1480
.   : milestone, 1456,
profiling (1.47 ms) : 1444, 1496
.   : milestone, 1470,
tracing (1.454 ms) : 1430, 1479
.   : milestone, 1454,
section candidate
no_agent (1.334 ms) : 1315, 1352
.   : milestone, 1334,
appsec (1.658 ms) : 1634, 1682
.   : milestone, 1658,
iast (1.457 ms) : 1434, 1481
.   : milestone, 1457,
profiling (1.475 ms) : 1449, 1502
.   : milestone, 1475,
tracing (1.472 ms) : 1449, 1496
.   : milestone, 1472,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.337 ms [1.317 ms, 1.356 ms] -
appsec 1.648 ms [1.624 ms, 1.673 ms] 311.95 µs (23.3%)
iast 1.456 ms [1.433 ms, 1.48 ms] 119.949 µs (9.0%)
profiling 1.47 ms [1.444 ms, 1.496 ms] 133.442 µs (10.0%)
tracing 1.454 ms [1.43 ms, 1.479 ms] 117.901 µs (8.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.334 ms [1.315 ms, 1.352 ms] -
appsec 1.658 ms [1.634 ms, 1.682 ms] 324.666 µs (24.3%)
iast 1.457 ms [1.434 ms, 1.481 ms] 123.904 µs (9.3%)
profiling 1.475 ms [1.449 ms, 1.502 ms] 141.616 µs (10.6%)
tracing 1.472 ms [1.449 ms, 1.496 ms] 138.725 µs (10.4%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~71a6ad3822, baseline=1.20.0-SNAPSHOT~9dd556a58b
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.946 µs) : 347, 387
.   : milestone, 367,
appsec (672.614 µs) : 652, 693
.   : milestone, 673,
iast (470.372 µs) : 449, 492
.   : milestone, 470,
iast_FULL (533.012 µs) : 512, 554
.   : milestone, 533,
iast_INACTIVE (445.788 µs) : 424, 467
.   : milestone, 446,
profiling (438.982 µs) : 418, 460
.   : milestone, 439,
tracing (446.17 µs) : 425, 468
.   : milestone, 446,
section candidate
no_agent (362.735 µs) : 343, 382
.   : milestone, 363,
appsec (669.604 µs) : 649, 690
.   : milestone, 670,
iast (465.292 µs) : 444, 486
.   : milestone, 465,
iast_FULL (534.797 µs) : 514, 555
.   : milestone, 535,
iast_INACTIVE (435.977 µs) : 415, 457
.   : milestone, 436,
profiling (451.221 µs) : 429, 473
.   : milestone, 451,
tracing (443.271 µs) : 422, 464
.   : milestone, 443,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 366.946 µs [346.586 µs, 387.306 µs] -
appsec 672.614 µs [651.922 µs, 693.305 µs] 305.667 µs (83.3%)
iast 470.372 µs [449.19 µs, 491.555 µs] 103.426 µs (28.2%)
iast_FULL 533.012 µs [512.216 µs, 553.808 µs] 166.066 µs (45.3%)
iast_INACTIVE 445.788 µs [424.467 µs, 467.109 µs] 78.842 µs (21.5%)
profiling 438.982 µs [418.177 µs, 459.786 µs] 72.036 µs (19.6%)
tracing 446.17 µs [424.645 µs, 467.695 µs] 79.224 µs (21.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.735 µs [343.393 µs, 382.078 µs] -
appsec 669.604 µs [648.819 µs, 690.388 µs] 306.868 µs (84.6%)
iast 465.292 µs [444.109 µs, 486.475 µs] 102.556 µs (28.3%)
iast_FULL 534.797 µs [514.307 µs, 555.287 µs] 172.062 µs (47.4%)
iast_INACTIVE 435.977 µs [415.432 µs, 456.522 µs] 73.241 µs (20.2%)
profiling 451.221 µs [429.488 µs, 472.954 µs] 88.485 µs (24.4%)
tracing 443.271 µs [422.293 µs, 464.249 µs] 80.536 µs (22.2%)

PerfectSlayer
PerfectSlayer reviewed Aug 10, 2023
View reviewed changes
Comment thread dd-java-agent/build.gradle
@smola
smola requested a review from PerfectSlayer August 14, 2023 06:51
smola added 3 commits August 17, 2023 10:07
@smola
Report dd-trace-java and its dependencies to telemetry
750b9a3 
While this might be sometimes confusing, we want to report the full
picture accurately.
@smola
Preserve pom.properties of our dependencies
18caf23
@smola
Set up shadowJar to fail on duplicate entries
71a6ad3 
This will ensure we do not reintroduce JAR corruption that was observed
in the past.
@smola
smola force-pushed the smola/report-tracer-deps branch from 1acd1d7 to 71a6ad3 Compare August 17, 2023 08:12
@smola
smola merged commit e306d30 into master Aug 17, 2023
@smola
smola deleted the smola/report-tracer-deps branch August 17, 2023 12:59
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 17, 2023
@ValentinZakharov ValentinZakharov added the tag: no release notes Changes to exclude from release notes label Aug 30, 2023
@smola smola removed the tag: no release notes Changes to exclude from release notes label Aug 31, 2023
@smola smola mentioned this pull request Sep 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes
@DDJavierSantos DDJavierSantos Awaiting requested review from DDJavierSantos DDJavierSantos was automatically assigned from DataDog/asm-java

Assignees

No one assigned

Labels

comp: telemetry Telemetry

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

3 participants

@smola @PerfectSlayer @ValentinZakharov