Reduce per-test code coverage overhead by nikita-tkachenko-datadog · Pull Request #6378 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2023-12-16T18:02:15Z

What Does This Do

Reduces the overhead of per-test code coverage in CI Visibility

pr-commenter · 2023-12-16T18:45:29Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikitat-tkachenko/per-test-coverage-optimisation
git_commit_date	1702652090	1702749602
git_commit_sha	`7a1a4ca`	`12e74d2`
release_version	1.26.0-SNAPSHOT~7a1a4cac29	1.26.0-SNAPSHOT~12e74d26c0

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702752286	1702752286
ci_job_id	391394408	391394408
ci_pipeline_id	25205776	25205776
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 5 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1052942
Total [baseline] (8.766 s) : 0, 8765947
Agent [candidate] (1.048 s) : 0, 1048130
Total [candidate] (8.738 s) : 0, 8738350
section iast
Agent [baseline] (1.166 s) : 0, 1166057
Total [baseline] (9.348 s) : 0, 9347947
Agent [candidate] (1.172 s) : 0, 1171753
Total [candidate] (9.293 s) : 0, 9292700
section iast_TELEMETRY_OFF
Agent [baseline] (1.158 s) : 0, 1158444
Total [baseline] (9.262 s) : 0, 9261886
Agent [candidate] (1.169 s) : 0, 1168907
Total [candidate] (9.275 s) : 0, 9275249

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	iast	1.166 s	113.115 ms (10.7%)
Agent	iast_TELEMETRY_OFF	1.158 s	105.502 ms (10.0%)
Total	tracing	8.766 s	-
Total	iast	9.348 s	582.0 ms (6.6%)
Total	iast_TELEMETRY_OFF	9.262 s	495.939 ms (5.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.048 s	-
Agent	iast	1.172 s	123.623 ms (11.8%)
Agent	iast_TELEMETRY_OFF	1.169 s	120.778 ms (11.5%)
Total	tracing	8.738 s	-
Total	iast	9.293 s	554.35 ms (6.3%)
Total	iast_TELEMETRY_OFF	9.275 s	536.898 ms (6.1%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (650.845 ms) : 0, 650845
BytebuddyAgent [candidate] (647.647 ms) : 0, 647647
GlobalTracer [baseline] (308.35 ms) : 0, 308350
GlobalTracer [candidate] (307.39 ms) : 0, 307390
AppSec [baseline] (51.299 ms) : 0, 51299
AppSec [candidate] (51.089 ms) : 0, 51089
Remote Config [baseline] (674.429 µs) : 0, 674
Remote Config [candidate] (669.614 µs) : 0, 670
Telemetry [baseline] (7.309 ms) : 0, 7309
Telemetry [candidate] (7.177 ms) : 0, 7177
section iast
BytebuddyAgent [baseline] (768.459 ms) : 0, 768459
BytebuddyAgent [candidate] (773.416 ms) : 0, 773416
GlobalTracer [baseline] (285.22 ms) : 0, 285220
GlobalTracer [candidate] (286.752 ms) : 0, 286752
AppSec [baseline] (49.217 ms) : 0, 49217
AppSec [candidate] (49.328 ms) : 0, 49328
IAST [baseline] (20.856 ms) : 0, 20856
IAST [candidate] (20.598 ms) : 0, 20598
Remote Config [baseline] (623.949 µs) : 0, 624
Remote Config [candidate] (604.894 µs) : 0, 605
Telemetry [baseline] (7.379 ms) : 0, 7379
Telemetry [candidate] (6.487 ms) : 0, 6487
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (761.072 ms) : 0, 761072
BytebuddyAgent [candidate] (767.977 ms) : 0, 767977
GlobalTracer [baseline] (285.602 ms) : 0, 285602
GlobalTracer [candidate] (288.303 ms) : 0, 288303
AppSec [baseline] (49.478 ms) : 0, 49478
AppSec [candidate] (49.965 ms) : 0, 49965
IAST [baseline] (20.856 ms) : 0, 20856
IAST [candidate] (20.824 ms) : 0, 20824
Remote Config [baseline] (621.372 µs) : 0, 621
Remote Config [candidate] (628.918 µs) : 0, 629
Telemetry [baseline] (6.632 ms) : 0, 6632
Telemetry [candidate] (6.647 ms) : 0, 6647

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.047 s) : 0, 1046804
Total [baseline] (9.381 s) : 0, 9380870
Agent [candidate] (1.054 s) : 0, 1053778
Total [candidate] (9.356 s) : 0, 9355827
section appsec
Agent [baseline] (1.147 s) : 0, 1146765
Total [baseline] (9.485 s) : 0, 9485376
Agent [candidate] (1.151 s) : 0, 1150875
Total [candidate] (9.485 s) : 0, 9485272
section iast
Agent [baseline] (1.164 s) : 0, 1164145
Total [baseline] (9.639 s) : 0, 9639021
Agent [candidate] (1.172 s) : 0, 1171963
Total [candidate] (9.585 s) : 0, 9584642
section profiling
Agent [baseline] (1.243 s) : 0, 1243062
Total [baseline] (9.679 s) : 0, 9679285
Agent [candidate] (1.255 s) : 0, 1254603
Total [candidate] (9.71 s) : 0, 9710191

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	appsec	1.147 s	99.961 ms (9.5%)
Agent	iast	1.164 s	117.341 ms (11.2%)
Agent	profiling	1.243 s	196.258 ms (18.7%)
Total	tracing	9.381 s	-
Total	appsec	9.485 s	104.506 ms (1.1%)
Total	iast	9.639 s	258.151 ms (2.8%)
Total	profiling	9.679 s	298.416 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.054 s	-
Agent	appsec	1.151 s	97.097 ms (9.2%)
Agent	iast	1.172 s	118.185 ms (11.2%)
Agent	profiling	1.255 s	200.825 ms (19.1%)
Total	tracing	9.356 s	-
Total	appsec	9.485 s	129.445 ms (1.4%)
Total	iast	9.585 s	228.815 ms (2.4%)
Total	profiling	9.71 s	354.364 ms (3.8%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (646.924 ms) : 0, 646924
BytebuddyAgent [candidate] (651.572 ms) : 0, 651572
GlobalTracer [baseline] (306.789 ms) : 0, 306789
GlobalTracer [candidate] (308.331 ms) : 0, 308331
AppSec [baseline] (51.023 ms) : 0, 51023
AppSec [candidate] (51.516 ms) : 0, 51516
Remote Config [baseline] (683.785 µs) : 0, 684
Remote Config [candidate] (676.727 µs) : 0, 677
Telemetry [baseline] (7.277 ms) : 0, 7277
Telemetry [candidate] (7.234 ms) : 0, 7234
section appsec
BytebuddyAgent [baseline] (648.257 ms) : 0, 648257
BytebuddyAgent [candidate] (650.982 ms) : 0, 650982
GlobalTracer [baseline] (306.883 ms) : 0, 306883
GlobalTracer [candidate] (308.548 ms) : 0, 308548
AppSec [baseline] (149.674 ms) : 0, 149674
AppSec [candidate] (149.539 ms) : 0, 149539
Remote Config [baseline] (649.358 µs) : 0, 649
Remote Config [candidate] (650.492 µs) : 0, 650
Telemetry [baseline] (6.869 ms) : 0, 6869
Telemetry [candidate] (6.861 ms) : 0, 6861
section iast
BytebuddyAgent [baseline] (768.011 ms) : 0, 768011
BytebuddyAgent [candidate] (774.978 ms) : 0, 774978
GlobalTracer [baseline] (284.573 ms) : 0, 284573
GlobalTracer [candidate] (286.159 ms) : 0, 286159
AppSec [baseline] (48.751 ms) : 0, 48751
AppSec [candidate] (49.03 ms) : 0, 49030
Remote Config [baseline] (611.696 µs) : 0, 612
Remote Config [candidate] (621.022 µs) : 0, 621
Telemetry [baseline] (7.298 ms) : 0, 7298
Telemetry [candidate] (6.596 ms) : 0, 6596
IAST [baseline] (20.561 ms) : 0, 20561
IAST [candidate] (19.908 ms) : 0, 19908
section profiling
ProfilingAgent [baseline] (94.421 ms) : 0, 94421
ProfilingAgent [candidate] (95.806 ms) : 0, 95806
BytebuddyAgent [baseline] (658.129 ms) : 0, 658129
BytebuddyAgent [candidate] (663.29 ms) : 0, 663290
GlobalTracer [baseline] (376.85 ms) : 0, 376850
GlobalTracer [candidate] (380.827 ms) : 0, 380827
AppSec [baseline] (51.415 ms) : 0, 51415
AppSec [candidate] (51.853 ms) : 0, 51853
Remote Config [baseline] (671.446 µs) : 0, 671
Remote Config [candidate] (668.806 µs) : 0, 669
Telemetry [baseline] (7.44 ms) : 0, 7440
Telemetry [candidate] (7.507 ms) : 0, 7507
Profiling [baseline] (94.446 ms) : 0, 94446
Profiling [candidate] (95.832 ms) : 0, 95832

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-16T18:23:56	2023-12-16T18:40:33
git_branch	master	nikitat-tkachenko/per-test-coverage-optimisation
git_commit_date	1702652090	1702749602
git_commit_sha	`7a1a4ca`	`12e74d2`
release_version	1.26.0-SNAPSHOT~7a1a4cac29	1.26.0-SNAPSHOT~12e74d26c0
start_time	2023-12-16T18:23:43	2023-12-16T18:40:20

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702752286	1702752286
ci_job_id	391394408	391394408
ci_pipeline_id	25205776	25205776
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29
    dateFormat X
    axisFormat %s
section baseline
no_agent (370.916 µs) : 350, 392
.   : milestone, 371,
iast (479.33 µs) : 459, 500
.   : milestone, 479,
iast_FULL (547.203 µs) : 527, 568
.   : milestone, 547,
iast_INACTIVE (443.468 µs) : 423, 464
.   : milestone, 443,
iast_TELEMETRY_OFF (468.882 µs) : 448, 490
.   : milestone, 469,
tracing (447.006 µs) : 426, 468
.   : milestone, 447,
section candidate
no_agent (367.739 µs) : 347, 388
.   : milestone, 368,
iast (474.449 µs) : 454, 495
.   : milestone, 474,
iast_FULL (539.791 µs) : 519, 560
.   : milestone, 540,
iast_INACTIVE (449.388 µs) : 429, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (467.438 µs) : 447, 488
.   : milestone, 467,
tracing (446.082 µs) : 425, 467
.   : milestone, 446,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	370.916 µs [349.7 µs, 392.133 µs]	-
iast	479.33 µs [459.152 µs, 499.508 µs]	108.414 µs (29.2%)
iast_FULL	547.203 µs [526.683 µs, 567.722 µs]	176.286 µs (47.5%)
iast_INACTIVE	443.468 µs [423.115 µs, 463.82 µs]	72.551 µs (19.6%)
iast_TELEMETRY_OFF	468.882 µs [448.213 µs, 489.551 µs]	97.966 µs (26.4%)
tracing	447.006 µs [426.147 µs, 467.865 µs]	76.09 µs (20.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.739 µs [347.442 µs, 388.037 µs]	-
iast	474.449 µs [454.077 µs, 494.821 µs]	106.71 µs (29.0%)
iast_FULL	539.791 µs [519.271 µs, 560.311 µs]	172.052 µs (46.8%)
iast_INACTIVE	449.388 µs [428.82 µs, 469.956 µs]	81.648 µs (22.2%)
iast_TELEMETRY_OFF	467.438 µs [446.915 µs, 487.961 µs]	99.698 µs (27.1%)
tracing	446.082 µs [425.125 µs, 467.04 µs]	78.343 µs (21.3%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~12e74d26c0, baseline=1.26.0-SNAPSHOT~7a1a4cac29
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.383 ms) : 1363, 1402
.   : milestone, 1383,
appsec (1.779 ms) : 1754, 1803
.   : milestone, 1779,
iast (1.551 ms) : 1527, 1575
.   : milestone, 1551,
profiling (1.53 ms) : 1503, 1557
.   : milestone, 1530,
tracing (1.485 ms) : 1460, 1511
.   : milestone, 1485,
section candidate
no_agent (1.346 ms) : 1327, 1366
.   : milestone, 1346,
appsec (1.779 ms) : 1754, 1804
.   : milestone, 1779,
iast (1.518 ms) : 1493, 1542
.   : milestone, 1518,
profiling (1.522 ms) : 1497, 1547
.   : milestone, 1522,
tracing (1.49 ms) : 1466, 1515
.   : milestone, 1490,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.383 ms [1.363 ms, 1.402 ms]	-
appsec	1.779 ms [1.754 ms, 1.803 ms]	396.019 µs (28.6%)
iast	1.551 ms [1.527 ms, 1.575 ms]	168.502 µs (12.2%)
profiling	1.53 ms [1.503 ms, 1.557 ms]	147.598 µs (10.7%)
tracing	1.485 ms [1.46 ms, 1.511 ms]	102.871 µs (7.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.346 ms [1.327 ms, 1.366 ms]	-
appsec	1.779 ms [1.754 ms, 1.804 ms]	432.404 µs (32.1%)
iast	1.518 ms [1.493 ms, 1.542 ms]	171.39 µs (12.7%)
profiling	1.522 ms [1.497 ms, 1.547 ms]	175.52 µs (13.0%)
tracing	1.49 ms [1.466 ms, 1.515 ms]	144.271 µs (10.7%)

Reduce per-test code coverage overhead

12e74d2

nikita-tkachenko-datadog added tag: performance Performance related changes comp: ci visibility Continuous Integration Visibility labels Dec 16, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 16, 2023 19:04

nikita-tkachenko-datadog requested a review from a team as a code owner December 16, 2023 19:04

nikita-tkachenko-datadog requested a review from juan-fernandez December 16, 2023 19:04

juan-fernandez approved these changes Dec 18, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit 6bb3505 into master Dec 18, 2023

nikita-tkachenko-datadog deleted the nikitat-tkachenko/per-test-coverage-optimisation branch December 18, 2023 10:29

github-actions Bot added this to the 1.26.0 milestone Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Reduce per-test code coverage overhead#6378

Reduce per-test code coverage overhead#6378
nikita-tkachenko-datadog merged 1 commit into
masterfrom
nikitat-tkachenko/per-test-coverage-optimisation

nikita-tkachenko-datadog commented Dec 16, 2023 •

edited by atlassian Bot

Loading

Uh oh!

pr-commenter Bot commented Dec 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nikita-tkachenko-datadog commented Dec 16, 2023 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Uh oh!

pr-commenter Bot commented Dec 16, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikita-tkachenko-datadog commented Dec 16, 2023 •

edited by atlassian Bot

Loading