Make Maven test module names unique by nikita-tkachenko-datadog · Pull Request #5762 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2023-08-23T09:59:11Z

What Does This Do

Changes the way CI Visibility test module names are calculated in instrumented Maven projects.
Previous module name used to include project name and lifecycle phase.
New module name includes project name, plugin name, and execution ID.

Motivation

Project name and lifecycle phase combination gives a module name that is not necessarily unique: it is possible to have a project configuration that binds multiple test executions in the same module to the same phase.
Project name, plugin name and execution ID combination is unique, Maven itself enforces this.

Additional Notes

Non-unique module names may cause the following issues:

if two modules with the same name execute in parallel the build will fail (due to some internal details of the test module registry implementation that uses the module name to uniquely identify DDTestModule instance)
it will be impossible to distinguish between different test executions in the UI

pr-commenter · 2023-08-23T10:37:05Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.20.0-SNAPSHOT~cbe55c2700	1.20.0-SNAPSHOT~f11e60c74c
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (998.038 ms) : 0, 998038
Total [baseline] (9.24 s) : 0, 9240114
Agent [candidate] (997.314 ms) : 0, 997314
Total [candidate] (9.218 s) : 0, 9217567
section appsec
Agent [baseline] (1.082 s) : 0, 1081861
Total [baseline] (9.368 s) : 0, 9368450
Agent [candidate] (1.079 s) : 0, 1079166
Total [candidate] (9.271 s) : 0, 9271414
section iast
Agent [baseline] (1.113 s) : 0, 1112674
Total [baseline] (9.512 s) : 0, 9512117
Agent [candidate] (1.092 s) : 0, 1091891
Total [candidate] (9.44 s) : 0, 9440258
section profiling
Agent [baseline] (1.165 s) : 0, 1164930
Total [baseline] (9.541 s) : 0, 9540723
Agent [candidate] (1.163 s) : 0, 1162686
Total [candidate] (9.479 s) : 0, 9479135

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	998.038 ms	-
Agent	appsec	1.082 s	83.823 ms (8.4%)
Agent	iast	1.113 s	114.636 ms (11.5%)
Agent	profiling	1.165 s	166.892 ms (16.7%)
Total	tracing	9.24 s	-
Total	appsec	9.368 s	128.336 ms (1.4%)
Total	iast	9.512 s	272.003 ms (2.9%)
Total	profiling	9.541 s	300.609 ms (3.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	997.314 ms	-
Agent	appsec	1.079 s	81.853 ms (8.2%)
Agent	iast	1.092 s	94.577 ms (9.5%)
Agent	profiling	1.163 s	165.373 ms (16.6%)
Total	tracing	9.218 s	-
Total	appsec	9.271 s	53.847 ms (0.6%)
Total	iast	9.44 s	222.691 ms (2.4%)
Total	profiling	9.479 s	261.569 ms (2.8%)

gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (620.209 ms) : 0, 620209
BytebuddyAgent [candidate] (617.841 ms) : 0, 617841
GlobalTracer [baseline] (286.218 ms) : 0, 286218
GlobalTracer [candidate] (287.799 ms) : 0, 287799
AppSec [baseline] (50.219 ms) : 0, 50219
AppSec [candidate] (50.271 ms) : 0, 50271
Remote Config [baseline] (667.781 µs) : 0, 668
Remote Config [candidate] (681.538 µs) : 0, 682
Telemetry [baseline] (6.039 ms) : 0, 6039
Telemetry [candidate] (6.046 ms) : 0, 6046
section appsec
BytebuddyAgent [baseline] (618.062 ms) : 0, 618062
BytebuddyAgent [candidate] (616.871 ms) : 0, 616871
GlobalTracer [baseline] (287.163 ms) : 0, 287163
GlobalTracer [candidate] (286.221 ms) : 0, 286221
AppSec [baseline] (135.315 ms) : 0, 135315
AppSec [candidate] (135.006 ms) : 0, 135006
Remote Config [baseline] (733.098 µs) : 0, 733
Remote Config [candidate] (748.946 µs) : 0, 749
Telemetry [baseline] (5.795 ms) : 0, 5795
Telemetry [candidate] (5.773 ms) : 0, 5773
section iast
BytebuddyAgent [baseline] (735.154 ms) : 0, 735154
BytebuddyAgent [candidate] (718.782 ms) : 0, 718782
GlobalTracer [baseline] (274.17 ms) : 0, 274170
GlobalTracer [candidate] (271.383 ms) : 0, 271383
AppSec [baseline] (47.868 ms) : 0, 47868
AppSec [candidate] (47.394 ms) : 0, 47394
IAST [baseline] (14.038 ms) : 0, 14038
IAST [candidate] (13.765 ms) : 0, 13765
Remote Config [baseline] (554.038 µs) : 0, 554
Remote Config [candidate] (546.05 µs) : 0, 546
Telemetry [baseline] (5.773 ms) : 0, 5773
Telemetry [candidate] (5.655 ms) : 0, 5655
section profiling
BytebuddyAgent [baseline] (629.999 ms) : 0, 629999
BytebuddyAgent [candidate] (628.89 ms) : 0, 628890
GlobalTracer [baseline] (348.675 ms) : 0, 348675
GlobalTracer [candidate] (347.782 ms) : 0, 347782
AppSec [baseline] (49.47 ms) : 0, 49470
AppSec [candidate] (49.316 ms) : 0, 49316
Remote Config [baseline] (661.337 µs) : 0, 661
Remote Config [candidate] (651.719 µs) : 0, 652
Telemetry [baseline] (6.021 ms) : 0, 6021
Telemetry [candidate] (6.043 ms) : 0, 6043
ProfilingAgent [baseline] (76.624 ms) : 0, 76624
ProfilingAgent [candidate] (76.753 ms) : 0, 76753
Profiling [baseline] (76.649 ms) : 0, 76649
Profiling [candidate] (76.778 ms) : 0, 76778

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (990.02 ms) : 0, 990020
Total [baseline] (8.68 s) : 0, 8679663
Agent [candidate] (987.755 ms) : 0, 987755
Total [candidate] (8.66 s) : 0, 8660202
section appsec
Agent [baseline] (1.073 s) : 0, 1073354
Total [baseline] (8.731 s) : 0, 8731137
Agent [candidate] (1.073 s) : 0, 1072518
Total [candidate] (8.721 s) : 0, 8720534
section iast
Agent [baseline] (1.09 s) : 0, 1090492
Total [baseline] (9.175 s) : 0, 9175218
Agent [candidate] (1.091 s) : 0, 1090699
Total [candidate] (9.163 s) : 0, 9162591
section profiling
Agent [baseline] (1.173 s) : 0, 1172571
Total [baseline] (8.958 s) : 0, 8957941
Agent [candidate] (1.164 s) : 0, 1164167
Total [candidate] (8.899 s) : 0, 8899337

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	990.02 ms	-
Agent	appsec	1.073 s	83.334 ms (8.4%)
Agent	iast	1.09 s	100.472 ms (10.1%)
Agent	profiling	1.173 s	182.551 ms (18.4%)
Total	tracing	8.68 s	-
Total	appsec	8.731 s	51.474 ms (0.6%)
Total	iast	9.175 s	495.555 ms (5.7%)
Total	profiling	8.958 s	278.278 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	987.755 ms	-
Agent	appsec	1.073 s	84.763 ms (8.6%)
Agent	iast	1.091 s	102.944 ms (10.4%)
Agent	profiling	1.164 s	176.412 ms (17.9%)
Total	tracing	8.66 s	-
Total	appsec	8.721 s	60.332 ms (0.7%)
Total	iast	9.163 s	502.388 ms (5.8%)
Total	profiling	8.899 s	239.135 ms (2.8%)

gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (613.371 ms) : 0, 613371
BytebuddyAgent [candidate] (612.397 ms) : 0, 612397
GlobalTracer [baseline] (285.107 ms) : 0, 285107
GlobalTracer [candidate] (284.158 ms) : 0, 284158
AppSec [baseline] (50.319 ms) : 0, 50319
AppSec [candidate] (50.262 ms) : 0, 50262
Remote Config [baseline] (676.146 µs) : 0, 676
Remote Config [candidate] (658.562 µs) : 0, 659
Telemetry [baseline] (6.085 ms) : 0, 6085
Telemetry [candidate] (5.966 ms) : 0, 5966
section appsec
BytebuddyAgent [baseline] (612.928 ms) : 0, 612928
BytebuddyAgent [candidate] (612.552 ms) : 0, 612552
GlobalTracer [baseline] (284.684 ms) : 0, 284684
GlobalTracer [candidate] (284.776 ms) : 0, 284776
AppSec [baseline] (134.868 ms) : 0, 134868
AppSec [candidate] (134.495 ms) : 0, 134495
Remote Config [baseline] (723.412 µs) : 0, 723
Remote Config [candidate] (726.485 µs) : 0, 726
Telemetry [baseline] (5.805 ms) : 0, 5805
Telemetry [candidate] (5.735 ms) : 0, 5735
section iast
BytebuddyAgent [baseline] (717.928 ms) : 0, 717928
BytebuddyAgent [candidate] (718.58 ms) : 0, 718580
GlobalTracer [baseline] (270.806 ms) : 0, 270806
GlobalTracer [candidate] (270.494 ms) : 0, 270494
AppSec [baseline] (47.279 ms) : 0, 47279
AppSec [candidate] (47.145 ms) : 0, 47145
IAST [baseline] (13.906 ms) : 0, 13906
IAST [candidate] (13.878 ms) : 0, 13878
Remote Config [baseline] (546.014 µs) : 0, 546
Remote Config [candidate] (559.553 µs) : 0, 560
Telemetry [baseline] (5.693 ms) : 0, 5693
Telemetry [candidate] (5.733 ms) : 0, 5733
section profiling
BytebuddyAgent [baseline] (633.703 ms) : 0, 633703
BytebuddyAgent [candidate] (629.73 ms) : 0, 629730
GlobalTracer [baseline] (350.03 ms) : 0, 350030
GlobalTracer [candidate] (347.899 ms) : 0, 347899
AppSec [baseline] (49.709 ms) : 0, 49709
AppSec [candidate] (49.069 ms) : 0, 49069
Remote Config [baseline] (660.28 µs) : 0, 660
Remote Config [candidate] (651.726 µs) : 0, 652
Telemetry [baseline] (6.085 ms) : 0, 6085
Telemetry [candidate] (6.028 ms) : 0, 6028
ProfilingAgent [baseline] (78.664 ms) : 0, 78664
ProfilingAgent [candidate] (77.615 ms) : 0, 77615
Profiling [baseline] (78.688 ms) : 0, 78688
Profiling [candidate] (77.639 ms) : 0, 77639

Load

Parameters

	Baseline	Candidate
commit	1.20.0-SNAPSHOT~cbe55c2700	1.20.0-SNAPSHOT~f11e60c74c
config	baseline	candidate
end_time	2023-08-23T10:13:46	2023-08-23T10:31:52
start_time	2023-08-23T10:13:29	2023-08-23T10:31:34

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.359 ms) : 1341, 1378
.   : milestone, 1359,
appsec (1.66 ms) : 1635, 1685
.   : milestone, 1660,
iast (1.442 ms) : 1416, 1467
.   : milestone, 1442,
profiling (1.471 ms) : 1445, 1497
.   : milestone, 1471,
tracing (1.468 ms) : 1444, 1492
.   : milestone, 1468,
section candidate
no_agent (1.338 ms) : 1319, 1357
.   : milestone, 1338,
appsec (1.686 ms) : 1662, 1710
.   : milestone, 1686,
iast (1.444 ms) : 1420, 1468
.   : milestone, 1444,
profiling (1.479 ms) : 1453, 1504
.   : milestone, 1479,
tracing (1.462 ms) : 1438, 1486
.   : milestone, 1462,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.359 ms [1.341 ms, 1.378 ms]	-
appsec	1.66 ms [1.635 ms, 1.685 ms]	300.396 µs (22.1%)
iast	1.442 ms [1.416 ms, 1.467 ms]	82.422 µs (6.1%)
profiling	1.471 ms [1.445 ms, 1.497 ms]	111.59 µs (8.2%)
tracing	1.468 ms [1.444 ms, 1.492 ms]	108.602 µs (8.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.338 ms [1.319 ms, 1.357 ms]	-
appsec	1.686 ms [1.662 ms, 1.71 ms]	348.129 µs (26.0%)
iast	1.444 ms [1.42 ms, 1.468 ms]	106.148 µs (7.9%)
profiling	1.479 ms [1.453 ms, 1.504 ms]	140.845 µs (10.5%)
tracing	1.462 ms [1.438 ms, 1.486 ms]	124.263 µs (9.3%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~f11e60c74c, baseline=1.20.0-SNAPSHOT~cbe55c2700
    dateFormat X
    axisFormat %s
section baseline
no_agent (362.895 µs) : 342, 383
.   : milestone, 363,
appsec (663.322 µs) : 643, 684
.   : milestone, 663,
iast (453.831 µs) : 433, 475
.   : milestone, 454,
iast_FULL (524.528 µs) : 504, 545
.   : milestone, 525,
iast_INACTIVE (434.563 µs) : 414, 455
.   : milestone, 435,
profiling (448.918 µs) : 426, 472
.   : milestone, 449,
tracing (429.531 µs) : 409, 450
.   : milestone, 430,
section candidate
no_agent (364.543 µs) : 343, 386
.   : milestone, 365,
appsec (663.298 µs) : 642, 684
.   : milestone, 663,
iast (453.121 µs) : 433, 474
.   : milestone, 453,
iast_FULL (527.476 µs) : 507, 548
.   : milestone, 527,
iast_INACTIVE (436.365 µs) : 416, 457
.   : milestone, 436,
profiling (440.639 µs) : 420, 461
.   : milestone, 441,
tracing (431.053 µs) : 410, 452
.   : milestone, 431,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	362.895 µs [342.383 µs, 383.406 µs]	-
appsec	663.322 µs [642.978 µs, 683.667 µs]	300.428 µs (82.8%)
iast	453.831 µs [432.873 µs, 474.789 µs]	90.937 µs (25.1%)
iast_FULL	524.528 µs [503.643 µs, 545.413 µs]	161.633 µs (44.5%)
iast_INACTIVE	434.563 µs [413.956 µs, 455.171 µs]	71.669 µs (19.7%)
profiling	448.918 µs [426.133 µs, 471.703 µs]	86.024 µs (23.7%)
tracing	429.531 µs [409.175 µs, 449.888 µs]	66.637 µs (18.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	364.543 µs [342.751 µs, 386.335 µs]	-
appsec	663.298 µs [642.392 µs, 684.203 µs]	298.755 µs (82.0%)
iast	453.121 µs [432.614 µs, 473.629 µs]	88.579 µs (24.3%)
iast_FULL	527.476 µs [506.525 µs, 548.427 µs]	162.933 µs (44.7%)
iast_INACTIVE	436.365 µs [415.659 µs, 457.071 µs]	71.822 µs (19.7%)
profiling	440.639 µs [419.871 µs, 461.407 µs]	76.096 µs (20.9%)
tracing	431.053 µs [410.061 µs, 452.046 µs]	66.511 µs (18.2%)

Make Maven test module names unique

f11e60c

nikita-tkachenko-datadog added type: bug fix Bug fix comp: ci visibility Continuous Integration Visibility labels Aug 23, 2023

nikita-tkachenko-datadog marked this pull request as ready for review August 23, 2023 10:54

nikita-tkachenko-datadog requested a review from a team as a code owner August 23, 2023 10:54

romainkomorn-exdatadog approved these changes Aug 23, 2023

View reviewed changes

nikita-tkachenko-datadog mentioned this pull request Aug 23, 2023

Make Maven test module names unique #5761

Merged

nikita-tkachenko-datadog merged commit 6d21c3e into master Aug 23, 2023

nikita-tkachenko-datadog deleted the nikita-tkachenko/maven-unique-module-name branch August 23, 2023 15:08

github-actions Bot added this to the 1.20.0 milestone Aug 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make Maven test module names unique#5762

Make Maven test module names unique#5762
nikita-tkachenko-datadog merged 1 commit into
masterfrom
nikita-tkachenko/maven-unique-module-name

nikita-tkachenko-datadog commented Aug 23, 2023

Uh oh!

pr-commenter Bot commented Aug 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nikita-tkachenko-datadog commented Aug 23, 2023

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Aug 23, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants