Implement flaky test retries for JUnit 5.3 by nikita-tkachenko-datadog · Pull Request #6332 · DataDog/dd-trace-java

nikita-tkachenko-datadog · 2023-12-07T20:21:30Z

What Does This Do

Implements retrying flaky Junit 5 tests.

Motivation

Adding flaky test retries feature to CI Visibility.

Jira ticket: CIVIS-8243

pr-commenter · 2023-12-07T21:04:28Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/retry-flaky-tests
git_commit_date	1702469793	1702473442
git_commit_sha	`b0cf14d`	`968fa35`
release_version	1.26.0-SNAPSHOT~b0cf14ddda	1.26.0-SNAPSHOT~968fa350a5

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702476284	1702476284
ci_job_id	388865440	388865440
ci_pipeline_id	25021888	25021888
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 6 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1043077
Total [baseline] (8.709 s) : 0, 8708533
Agent [candidate] (1.046 s) : 0, 1046178
Total [candidate] (8.711 s) : 0, 8711088
section iast
Agent [baseline] (1.163 s) : 0, 1162931
Total [baseline] (9.315 s) : 0, 9315109
Agent [candidate] (1.171 s) : 0, 1171112
Total [candidate] (9.266 s) : 0, 9265946
section iast_TELEMETRY_OFF
Agent [baseline] (1.155 s) : 0, 1154543
Total [baseline] (9.24 s) : 0, 9240027
Agent [candidate] (1.172 s) : 0, 1171934
Total [candidate] (9.276 s) : 0, 9275936

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.163 s	119.854 ms (11.5%)
Agent	iast_TELEMETRY_OFF	1.155 s	111.466 ms (10.7%)
Total	tracing	8.709 s	-
Total	iast	9.315 s	606.576 ms (7.0%)
Total	iast_TELEMETRY_OFF	9.24 s	531.494 ms (6.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	iast	1.171 s	124.934 ms (11.9%)
Agent	iast_TELEMETRY_OFF	1.172 s	125.756 ms (12.0%)
Total	tracing	8.711 s	-
Total	iast	9.266 s	554.858 ms (6.4%)
Total	iast_TELEMETRY_OFF	9.276 s	564.848 ms (6.5%)

gantt
    title insecure-bank - break down per module: candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.675 ms) : 0, 645675
BytebuddyAgent [candidate] (647.736 ms) : 0, 647736
GlobalTracer [baseline] (306.504 ms) : 0, 306504
GlobalTracer [candidate] (307.813 ms) : 0, 307813
AppSec [baseline] (48.808 ms) : 0, 48808
AppSec [candidate] (48.517 ms) : 0, 48517
Remote Config [baseline] (674.712 µs) : 0, 675
Remote Config [candidate] (677.736 µs) : 0, 678
Telemetry [baseline] (7.128 ms) : 0, 7128
Telemetry [candidate] (7.133 ms) : 0, 7133
section iast
BytebuddyAgent [baseline] (768.396 ms) : 0, 768396
BytebuddyAgent [candidate] (773.105 ms) : 0, 773105
GlobalTracer [baseline] (285.648 ms) : 0, 285648
GlobalTracer [candidate] (287.145 ms) : 0, 287145
AppSec [baseline] (46.966 ms) : 0, 46966
AppSec [candidate] (47.43 ms) : 0, 47430
IAST [baseline] (20.371 ms) : 0, 20371
IAST [candidate] (21.735 ms) : 0, 21735
Remote Config [baseline] (607.257 µs) : 0, 607
Remote Config [candidate] (597.672 µs) : 0, 598
Telemetry [baseline] (6.638 ms) : 0, 6638
Telemetry [candidate] (6.532 ms) : 0, 6532
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (760.07 ms) : 0, 760070
BytebuddyAgent [candidate] (772.617 ms) : 0, 772617
GlobalTracer [baseline] (286.041 ms) : 0, 286041
GlobalTracer [candidate] (289.231 ms) : 0, 289231
AppSec [baseline] (46.218 ms) : 0, 46218
AppSec [candidate] (47.155 ms) : 0, 47155
IAST [baseline] (19.216 ms) : 0, 19216
IAST [candidate] (19.443 ms) : 0, 19443
Remote Config [baseline] (599.607 µs) : 0, 600
Remote Config [candidate] (2.06 ms) : 0, 2060
Telemetry [baseline] (8.054 ms) : 0, 8054
Telemetry [candidate] (6.691 ms) : 0, 6691

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1056065
Total [baseline] (9.403 s) : 0, 9402671
Agent [candidate] (1.046 s) : 0, 1045992
Total [candidate] (9.367 s) : 0, 9366690
section appsec
Agent [baseline] (1.131 s) : 0, 1131334
Total [baseline] (9.418 s) : 0, 9418049
Agent [candidate] (1.132 s) : 0, 1132441
Total [candidate] (9.421 s) : 0, 9420866
section iast
Agent [baseline] (1.158 s) : 0, 1158419
Total [baseline] (9.603 s) : 0, 9603007
Agent [candidate] (1.172 s) : 0, 1172112
Total [candidate] (9.588 s) : 0, 9588111
section profiling
Agent [baseline] (1.231 s) : 0, 1231191
Total [baseline] (9.62 s) : 0, 9620167
Agent [candidate] (1.236 s) : 0, 1235523
Total [candidate] (9.637 s) : 0, 9637393

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.056 s	-
Agent	appsec	1.131 s	75.269 ms (7.1%)
Agent	iast	1.158 s	102.354 ms (9.7%)
Agent	profiling	1.231 s	175.125 ms (16.6%)
Total	tracing	9.403 s	-
Total	appsec	9.418 s	15.378 ms (0.2%)
Total	iast	9.603 s	200.335 ms (2.1%)
Total	profiling	9.62 s	217.495 ms (2.3%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	appsec	1.132 s	86.449 ms (8.3%)
Agent	iast	1.172 s	126.12 ms (12.1%)
Agent	profiling	1.236 s	189.531 ms (18.1%)
Total	tracing	9.367 s	-
Total	appsec	9.421 s	54.176 ms (0.6%)
Total	iast	9.588 s	221.421 ms (2.4%)
Total	profiling	9.637 s	270.702 ms (2.9%)

gantt
    title petclinic - break down per module: candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (654.461 ms) : 0, 654461
BytebuddyAgent [candidate] (647.839 ms) : 0, 647839
GlobalTracer [baseline] (310.15 ms) : 0, 310150
GlobalTracer [candidate] (307.461 ms) : 0, 307461
AppSec [baseline] (48.878 ms) : 0, 48878
AppSec [candidate] (48.633 ms) : 0, 48633
Remote Config [baseline] (685.504 µs) : 0, 686
Remote Config [candidate] (671.501 µs) : 0, 672
Telemetry [baseline] (7.229 ms) : 0, 7229
Telemetry [candidate] (7.109 ms) : 0, 7109
section appsec
BytebuddyAgent [baseline] (645.939 ms) : 0, 645939
BytebuddyAgent [candidate] (647.186 ms) : 0, 647186
GlobalTracer [baseline] (306.954 ms) : 0, 306954
GlobalTracer [candidate] (306.784 ms) : 0, 306784
AppSec [baseline] (136.784 ms) : 0, 136784
AppSec [candidate] (136.632 ms) : 0, 136632
Remote Config [baseline] (647.678 µs) : 0, 648
Remote Config [candidate] (641.275 µs) : 0, 641
Telemetry [baseline] (6.817 ms) : 0, 6817
Telemetry [candidate] (6.819 ms) : 0, 6819
section iast
BytebuddyAgent [baseline] (765.404 ms) : 0, 765404
BytebuddyAgent [candidate] (774.689 ms) : 0, 774689
GlobalTracer [baseline] (284.562 ms) : 0, 284562
GlobalTracer [candidate] (287.259 ms) : 0, 287259
AppSec [baseline] (46.83 ms) : 0, 46830
AppSec [candidate] (47.142 ms) : 0, 47142
Remote Config [baseline] (609.325 µs) : 0, 609
Remote Config [candidate] (623.462 µs) : 0, 623
Telemetry [baseline] (8.066 ms) : 0, 8066
Telemetry [candidate] (8.115 ms) : 0, 8115
IAST [baseline] (18.752 ms) : 0, 18752
IAST [candidate] (19.699 ms) : 0, 19699
section profiling
BytebuddyAgent [baseline] (656.384 ms) : 0, 656384
BytebuddyAgent [candidate] (659.926 ms) : 0, 659926
GlobalTracer [baseline] (376.179 ms) : 0, 376179
GlobalTracer [candidate] (376.898 ms) : 0, 376898
AppSec [baseline] (48.59 ms) : 0, 48590
AppSec [candidate] (48.41 ms) : 0, 48410
Remote Config [baseline] (678.764 µs) : 0, 679
Remote Config [candidate] (686.01 µs) : 0, 686
Telemetry [baseline] (7.458 ms) : 0, 7458
Telemetry [candidate] (7.42 ms) : 0, 7420
ProfilingAgent [baseline] (87.849 ms) : 0, 87849
ProfilingAgent [candidate] (87.991 ms) : 0, 87991
Profiling [baseline] (87.874 ms) : 0, 87874
Profiling [candidate] (88.017 ms) : 0, 88017

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2023-12-13T13:43:58	2023-12-13T14:00:30
git_branch	master	nikita-tkachenko/retry-flaky-tests
git_commit_date	1702469793	1702473442
git_commit_sha	`b0cf14d`	`968fa35`
release_version	1.26.0-SNAPSHOT~b0cf14ddda	1.26.0-SNAPSHOT~968fa350a5
start_time	2023-12-13T13:43:44	2023-12-13T14:00:17

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1702476284	1702476284
ci_job_id	388865440	388865440
ci_pipeline_id	25021888	25021888
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.435 µs) : 346, 386
.   : milestone, 366,
iast (480.251 µs) : 460, 501
.   : milestone, 480,
iast_FULL (532.15 µs) : 512, 552
.   : milestone, 532,
iast_INACTIVE (440.397 µs) : 419, 461
.   : milestone, 440,
iast_TELEMETRY_OFF (467.32 µs) : 446, 488
.   : milestone, 467,
tracing (437.445 µs) : 417, 458
.   : milestone, 437,
section candidate
no_agent (362.474 µs) : 342, 383
.   : milestone, 362,
iast (471.727 µs) : 451, 492
.   : milestone, 472,
iast_FULL (542.859 µs) : 522, 564
.   : milestone, 543,
iast_INACTIVE (441.482 µs) : 421, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (468.106 µs) : 447, 489
.   : milestone, 468,
tracing (438.208 µs) : 418, 459
.   : milestone, 438,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	366.435 µs [346.437 µs, 386.432 µs]	-
iast	480.251 µs [459.885 µs, 500.617 µs]	113.816 µs (31.1%)
iast_FULL	532.15 µs [511.847 µs, 552.453 µs]	165.715 µs (45.2%)
iast_INACTIVE	440.397 µs [419.422 µs, 461.372 µs]	73.962 µs (20.2%)
iast_TELEMETRY_OFF	467.32 µs [446.204 µs, 488.435 µs]	100.885 µs (27.5%)
tracing	437.445 µs [416.527 µs, 458.364 µs]	71.011 µs (19.4%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	362.474 µs [342.262 µs, 382.685 µs]	-
iast	471.727 µs [451.3 µs, 492.154 µs]	109.253 µs (30.1%)
iast_FULL	542.859 µs [522.115 µs, 563.602 µs]	180.385 µs (49.8%)
iast_INACTIVE	441.482 µs [420.85 µs, 462.113 µs]	79.008 µs (21.8%)
iast_TELEMETRY_OFF	468.106 µs [447.436 µs, 488.777 µs]	105.633 µs (29.1%)
tracing	438.208 µs [417.712 µs, 458.703 µs]	75.734 µs (20.9%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.26.0-SNAPSHOT~968fa350a5, baseline=1.26.0-SNAPSHOT~b0cf14ddda
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.354 ms) : 1335, 1373
.   : milestone, 1354,
appsec (1.752 ms) : 1727, 1778
.   : milestone, 1752,
iast (1.529 ms) : 1504, 1553
.   : milestone, 1529,
profiling (1.52 ms) : 1494, 1546
.   : milestone, 1520,
tracing (1.502 ms) : 1476, 1527
.   : milestone, 1502,
section candidate
no_agent (1.363 ms) : 1344, 1382
.   : milestone, 1363,
appsec (1.752 ms) : 1726, 1777
.   : milestone, 1752,
iast (1.551 ms) : 1526, 1575
.   : milestone, 1551,
profiling (1.515 ms) : 1490, 1540
.   : milestone, 1515,
tracing (1.519 ms) : 1495, 1544
.   : milestone, 1519,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.354 ms [1.335 ms, 1.373 ms]	-
appsec	1.752 ms [1.727 ms, 1.778 ms]	398.951 µs (29.5%)
iast	1.529 ms [1.504 ms, 1.553 ms]	175.34 µs (13.0%)
profiling	1.52 ms [1.494 ms, 1.546 ms]	166.255 µs (12.3%)
tracing	1.502 ms [1.476 ms, 1.527 ms]	148.049 µs (10.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.363 ms [1.344 ms, 1.382 ms]	-
appsec	1.752 ms [1.726 ms, 1.777 ms]	388.513 µs (28.5%)
iast	1.551 ms [1.526 ms, 1.575 ms]	187.544 µs (13.8%)
profiling	1.515 ms [1.49 ms, 1.54 ms]	152.128 µs (11.2%)
tracing	1.519 ms [1.495 ms, 1.544 ms]	156.442 µs (11.5%)

bantonsson · 2023-12-13T15:18:00Z


  @Override
  public Object putIfAbsent(final Object key, final Factory<Object> contextFactory) {
+    return computeIfAbsent(key, k -> contextFactory.create());


Won't this create a holder for the contextFactory and this lambda on every call? Maybe having Factory implement KeyAwareFactory with a defender method would be better?

Good catch! And apologies, I completely forgot that this PR contained a change that needs to be reviewed by the core team (should've done that in a separate PR actually).
The fix is in #6360

nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Dec 7, 2023

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests branch from 41c75d4 to 18c21ce Compare December 9, 2023 16:16

nikita-tkachenko-datadog added the run-tests: all Run all tests label Dec 10, 2023

nikita-tkachenko-datadog marked this pull request as ready for review December 10, 2023 16:03

nikita-tkachenko-datadog requested review from a team as code owners December 10, 2023 16:03

nikita-tkachenko-datadog requested review from juan-fernandez, mcculls and ygree December 10, 2023 16:03

romainkomorn-exdatadog approved these changes Dec 13, 2023

View reviewed changes

nikita-tkachenko-datadog added 2 commits December 13, 2023 12:59

Implement flaky test retries for JUnit 5.3

284add4

Fix CI failures

4ca62c0

nikita-tkachenko-datadog force-pushed the nikita-tkachenko/retry-flaky-tests branch from a5d9bda to 4ca62c0 Compare December 13, 2023 12:00

Resolve merge conflicts

968fa35

nikita-tkachenko-datadog merged commit 30789d4 into master Dec 13, 2023

nikita-tkachenko-datadog deleted the nikita-tkachenko/retry-flaky-tests branch December 13, 2023 14:59

github-actions Bot added this to the 1.26.0 milestone Dec 13, 2023

bantonsson reviewed Dec 13, 2023

View reviewed changes

nikita-tkachenko-datadog mentioned this pull request Dec 13, 2023

Fix ContextStore#putIfAbsent method to not use a lambda #6360

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implement flaky test retries for JUnit 5.3#6332

Implement flaky test retries for JUnit 5.3#6332
nikita-tkachenko-datadog merged 3 commits into
masterfrom
nikita-tkachenko/retry-flaky-tests

nikita-tkachenko-datadog commented Dec 7, 2023 •

edited by atlassian Bot

Loading

Uh oh!

pr-commenter Bot commented Dec 7, 2023 •

edited

Loading

Uh oh!

bantonsson Dec 13, 2023 •

edited

Loading

Uh oh!

nikita-tkachenko-datadog Dec 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

nikita-tkachenko-datadog commented Dec 7, 2023 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Uh oh!

pr-commenter Bot commented Dec 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

bantonsson Dec 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nikita-tkachenko-datadog Dec 13, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nikita-tkachenko-datadog commented Dec 7, 2023 •

edited by atlassian Bot

Loading

pr-commenter Bot commented Dec 7, 2023 •

edited

Loading

bantonsson Dec 13, 2023 •

edited

Loading