Ignore configuration blocks when comparing skippable test identifiers by nikita-tkachenko-datadog · Pull Request #6379 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2023-12-16T18:40:31Z

What Does This Do

Ignores configuration objects when comparing TestIdentifier objects.

Motivation

Test identifiers comparison is done when determining whether to skip a test with ITR, or whether to retry a flaky test.
There are some inconsistencies in how the backend populates test configurations (depending on some conditions it may return a partially filled configuration block, an empty object, or a null).
Rather than trying to replicate the backend's logic in the tracer, it was decided to not use test configurations for comparison: the relevant fields are not be present in the backend's response anyway (since they are provided by the tracer when requesting tests data, the backend does not include them in the response).

Jira ticket: CIVIS-8358

pr-commenter · 2023-12-16T19:41:55Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/itr-configurations-fix
git_commit_date	1702752103	1702751701
git_commit_sha	`61aca5c`	`2419872`
release_version	1.26.0-SNAPSHOT~61aca5c9d8	1.26.0-SNAPSHOT~24198723c4

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702755673	1702755673
ci_job_id	391398491	391398491
ci_pipeline_id	25205932	25205932
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 47 metrics, 7 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.048 s) : 0, 1047659
Total [baseline] (9.318 s) : 0, 9318442
Agent [candidate] (1.053 s) : 0, 1052967
Total [candidate] (9.381 s) : 0, 9381134
section appsec
Agent [baseline] (1.149 s) : 0, 1149355
Total [baseline] (9.558 s) : 0, 9558354
Agent [candidate] (1.142 s) : 0, 1142064
Total [candidate] (9.459 s) : 0, 9458626
section iast
Agent [baseline] (1.163 s) : 0, 1163067
Total [baseline] (9.519 s) : 0, 9518665
Agent [candidate] (1.164 s) : 0, 1164380
Total [candidate] (9.573 s) : 0, 9573130
section profiling
Agent [baseline] (1.241 s) : 0, 1241135
Total [baseline] (9.585 s) : 0, 9585416
Agent [candidate] (1.259 s) : 0, 1258828
Total [candidate] (9.573 s) : 0, 9572512

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.048 s	-
Agent	appsec	1.149 s	101.696 ms (9.7%)
Agent	iast	1.163 s	115.408 ms (11.0%)
Agent	profiling	1.241 s	193.476 ms (18.5%)
Total	tracing	9.318 s	-
Total	appsec	9.558 s	239.912 ms (2.6%)
Total	iast	9.519 s	200.223 ms (2.1%)
Total	profiling	9.585 s	266.974 ms (2.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.053 s	-
Agent	appsec	1.142 s	89.097 ms (8.5%)
Agent	iast	1.164 s	111.413 ms (10.6%)
Agent	profiling	1.259 s	205.861 ms (19.6%)
Total	tracing	9.381 s	-
Total	appsec	9.459 s	77.491 ms (0.8%)
Total	iast	9.573 s	191.996 ms (2.0%)
Total	profiling	9.573 s	191.378 ms (2.0%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (647.806 ms) : 0, 647806
BytebuddyAgent [candidate] (651.086 ms) : 0, 651086
GlobalTracer [baseline] (306.66 ms) : 0, 306660
GlobalTracer [candidate] (308.547 ms) : 0, 308547
AppSec [baseline] (50.96 ms) : 0, 50960
AppSec [candidate] (51.001 ms) : 0, 51001
Remote Config [baseline] (681.143 µs) : 0, 681
Remote Config [candidate] (682.019 µs) : 0, 682
Telemetry [baseline] (7.285 ms) : 0, 7285
Telemetry [candidate] (7.285 ms) : 0, 7285
section appsec
BytebuddyAgent [baseline] (650.278 ms) : 0, 650278
BytebuddyAgent [candidate] (645.995 ms) : 0, 645995
GlobalTracer [baseline] (307.33 ms) : 0, 307330
GlobalTracer [candidate] (305.588 ms) : 0, 305588
AppSec [baseline] (149.239 ms) : 0, 149239
AppSec [candidate] (148.987 ms) : 0, 148987
Remote Config [baseline] (654.893 µs) : 0, 655
Remote Config [candidate] (642.333 µs) : 0, 642
Telemetry [baseline] (7.467 ms) : 0, 7467
Telemetry [candidate] (6.791 ms) : 0, 6791
section iast
BytebuddyAgent [baseline] (767.055 ms) : 0, 767055
BytebuddyAgent [candidate] (767.884 ms) : 0, 767884
GlobalTracer [baseline] (284.564 ms) : 0, 284564
GlobalTracer [candidate] (285.308 ms) : 0, 285308
AppSec [baseline] (48.86 ms) : 0, 48860
AppSec [candidate] (48.846 ms) : 0, 48846
Remote Config [baseline] (607.564 µs) : 0, 608
Remote Config [candidate] (611.142 µs) : 0, 611
Telemetry [baseline] (7.23 ms) : 0, 7230
Telemetry [candidate] (6.461 ms) : 0, 6461
IAST [baseline] (20.502 ms) : 0, 20502
IAST [candidate] (21.062 ms) : 0, 21062
section profiling
ProfilingAgent [baseline] (94.154 ms) : 0, 94154
ProfilingAgent [candidate] (95.272 ms) : 0, 95272
BytebuddyAgent [baseline] (656.86 ms) : 0, 656860
BytebuddyAgent [candidate] (666.961 ms) : 0, 666961
GlobalTracer [baseline] (376.458 ms) : 0, 376458
GlobalTracer [candidate] (381.542 ms) : 0, 381542
AppSec [baseline] (51.512 ms) : 0, 51512
AppSec [candidate] (51.965 ms) : 0, 51965
Remote Config [baseline] (662.459 µs) : 0, 662
Remote Config [candidate] (672.588 µs) : 0, 673
Telemetry [baseline] (7.428 ms) : 0, 7428
Telemetry [candidate] (7.567 ms) : 0, 7567
Profiling [baseline] (94.18 ms) : 0, 94180
Profiling [candidate] (95.297 ms) : 0, 95297

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054536
Total [baseline] (8.712 s) : 0, 8712175
Agent [candidate] (1.056 s) : 0, 1055704
Total [candidate] (8.727 s) : 0, 8727212
section iast
Agent [baseline] (1.172 s) : 0, 1171502
Total [baseline] (9.269 s) : 0, 9268989
Agent [candidate] (1.165 s) : 0, 1164881
Total [candidate] (9.24 s) : 0, 9240130
section iast_TELEMETRY_OFF
Agent [baseline] (1.154 s) : 0, 1154356
Total [baseline] (9.228 s) : 0, 9228059
Agent [candidate] (1.169 s) : 0, 1168644
Total [candidate] (9.261 s) : 0, 9261478

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	iast	1.172 s	116.966 ms (11.1%)
Agent	iast_TELEMETRY_OFF	1.154 s	99.82 ms (9.5%)
Total	tracing	8.712 s	-
Total	iast	9.269 s	556.814 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.228 s	515.884 ms (5.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	iast	1.165 s	109.176 ms (10.3%)
Agent	iast_TELEMETRY_OFF	1.169 s	112.94 ms (10.7%)
Total	tracing	8.727 s	-
Total	iast	9.24 s	512.919 ms (5.9%)
Total	iast_TELEMETRY_OFF	9.261 s	534.266 ms (6.1%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (651.748 ms) : 0, 651748
BytebuddyAgent [candidate] (651.839 ms) : 0, 651839
GlobalTracer [baseline] (308.404 ms) : 0, 308404
GlobalTracer [candidate] (309.937 ms) : 0, 309937
AppSec [baseline] (51.888 ms) : 0, 51888
AppSec [candidate] (51.525 ms) : 0, 51525
Remote Config [baseline] (674.078 µs) : 0, 674
Remote Config [candidate] (677.953 µs) : 0, 678
Telemetry [baseline] (7.275 ms) : 0, 7275
Telemetry [candidate] (7.288 ms) : 0, 7288
section iast
BytebuddyAgent [baseline] (772.628 ms) : 0, 772628
BytebuddyAgent [candidate] (768.192 ms) : 0, 768192
GlobalTracer [baseline] (286.354 ms) : 0, 286354
GlobalTracer [candidate] (285.347 ms) : 0, 285347
AppSec [baseline] (49.396 ms) : 0, 49396
AppSec [candidate] (48.875 ms) : 0, 48875
Remote Config [baseline] (615.673 µs) : 0, 616
Remote Config [candidate] (611.134 µs) : 0, 611
Telemetry [baseline] (7.408 ms) : 0, 7408
Telemetry [candidate] (6.488 ms) : 0, 6488
IAST [baseline] (20.632 ms) : 0, 20632
IAST [candidate] (21.188 ms) : 0, 21188
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (758.879 ms) : 0, 758879
BytebuddyAgent [candidate] (768.166 ms) : 0, 768166
GlobalTracer [baseline] (284.538 ms) : 0, 284538
GlobalTracer [candidate] (288.252 ms) : 0, 288252
AppSec [baseline] (49.318 ms) : 0, 49318
AppSec [candidate] (49.565 ms) : 0, 49565
Remote Config [baseline] (625.821 µs) : 0, 626
Remote Config [candidate] (621.206 µs) : 0, 621
Telemetry [baseline] (7.356 ms) : 0, 7356
Telemetry [candidate] (8.81 ms) : 0, 8810
IAST [baseline] (19.493 ms) : 0, 19493
IAST [candidate] (18.665 ms) : 0, 18665

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-16T19:20:30	2023-12-16T19:37:02
git_branch	master	nikita-tkachenko/itr-configurations-fix
git_commit_date	1702752103	1702751701
git_commit_sha	`61aca5c`	`2419872`
release_version	1.26.0-SNAPSHOT~61aca5c9d8	1.26.0-SNAPSHOT~24198723c4
start_time	2023-12-16T19:20:17	2023-12-16T19:36:49

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702755673	1702755673
ci_job_id	391398491	391398491
ci_pipeline_id	25205932	25205932
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.346 ms) : 1327, 1365
.   : milestone, 1346,
appsec (1.753 ms) : 1728, 1777
.   : milestone, 1753,
iast (1.526 ms) : 1502, 1550
.   : milestone, 1526,
profiling (1.514 ms) : 1489, 1540
.   : milestone, 1514,
tracing (1.494 ms) : 1469, 1519
.   : milestone, 1494,
section candidate
no_agent (1.35 ms) : 1332, 1369
.   : milestone, 1350,
appsec (1.737 ms) : 1712, 1763
.   : milestone, 1737,
iast (1.505 ms) : 1481, 1530
.   : milestone, 1505,
profiling (1.497 ms) : 1472, 1522
.   : milestone, 1497,
tracing (1.502 ms) : 1477, 1526
.   : milestone, 1502,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.346 ms [1.327 ms, 1.365 ms]	-
appsec	1.753 ms [1.728 ms, 1.777 ms]	407.023 µs (30.2%)
iast	1.526 ms [1.502 ms, 1.55 ms]	180.387 µs (13.4%)
profiling	1.514 ms [1.489 ms, 1.54 ms]	168.561 µs (12.5%)
tracing	1.494 ms [1.469 ms, 1.519 ms]	148.198 µs (11.0%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.35 ms [1.332 ms, 1.369 ms]	-
appsec	1.737 ms [1.712 ms, 1.763 ms]	386.881 µs (28.6%)
iast	1.505 ms [1.481 ms, 1.53 ms]	154.824 µs (11.5%)
profiling	1.497 ms [1.472 ms, 1.522 ms]	146.709 µs (10.9%)
tracing	1.502 ms [1.477 ms, 1.526 ms]	151.154 µs (11.2%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~24198723c4, baseline=1.26.0-SNAPSHOT~61aca5c9d8
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.021 µs) : 345, 389
.   : milestone, 367,
iast (460.899 µs) : 441, 481
.   : milestone, 461,
iast_FULL (533.975 µs) : 514, 554
.   : milestone, 534,
iast_INACTIVE (439.477 µs) : 419, 460
.   : milestone, 439,
iast_TELEMETRY_OFF (473.719 µs) : 453, 495
.   : milestone, 474,
tracing (437.941 µs) : 416, 460
.   : milestone, 438,
section candidate
no_agent (360.853 µs) : 340, 381
.   : milestone, 361,
iast (470.507 µs) : 450, 491
.   : milestone, 471,
iast_FULL (531.613 µs) : 511, 552
.   : milestone, 532,
iast_INACTIVE (445.292 µs) : 424, 466
.   : milestone, 445,
iast_TELEMETRY_OFF (455.098 µs) : 435, 475
.   : milestone, 455,
tracing (435.569 µs) : 415, 457
.   : milestone, 436,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.021 µs [345.12 µs, 388.923 µs]	-
iast	460.899 µs [440.681 µs, 481.116 µs]	93.878 µs (25.6%)
iast_FULL	533.975 µs [513.66 µs, 554.29 µs]	166.954 µs (45.5%)
iast_INACTIVE	439.477 µs [419.035 µs, 459.92 µs]	72.456 µs (19.7%)
iast_TELEMETRY_OFF	473.719 µs [452.897 µs, 494.54 µs]	106.697 µs (29.1%)
tracing	437.941 µs [416.308 µs, 459.574 µs]	70.92 µs (19.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	360.853 µs [340.33 µs, 381.377 µs]	-
iast	470.507 µs [450.273 µs, 490.741 µs]	109.654 µs (30.4%)
iast_FULL	531.613 µs [511.104 µs, 552.122 µs]	170.76 µs (47.3%)
iast_INACTIVE	445.292 µs [424.115 µs, 466.468 µs]	84.439 µs (23.4%)
iast_TELEMETRY_OFF	455.098 µs [434.794 µs, 475.402 µs]	94.245 µs (26.1%)
tracing	435.569 µs [414.604 µs, 456.535 µs]	74.716 µs (20.7%)

Ignore configurations block when comparing skippable test identifiers

2419872

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 16, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 16, 2023 19:45

nikita-tkachenko-datadog requested a review from a team as a code owner December 16, 2023 19:45

nikita-tkachenko-datadog requested a review from romainkomorn-exdatadog December 16, 2023 19:45

romainkomorn-exdatadog approved these changes Dec 18, 2023

View reviewed changes

nikita-tkachenko-datadog merged commit b820c11 into master Dec 18, 2023

nikita-tkachenko-datadog deleted the nikita-tkachenko/itr-configurations-fix branch December 18, 2023 10:29

github-actions Bot added this to the 1.26.0 milestone Dec 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ignore configuration blocks when comparing skippable test identifiers#6379

Ignore configuration blocks when comparing skippable test identifiers#6379
nikita-tkachenko-datadog merged 1 commit into
masterfrom
nikita-tkachenko/itr-configurations-fix

nikita-tkachenko-datadog commented Dec 16, 2023 •

edited by atlassian Bot

Loading

Uh oh!

pr-commenter Bot commented Dec 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

nikita-tkachenko-datadog commented Dec 16, 2023 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Uh oh!

pr-commenter Bot commented Dec 16, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nikita-tkachenko-datadog commented Dec 16, 2023 •

edited by atlassian Bot

Loading