Skip to content

Mark Vert.X Redis Client 4 integration test as flaky#6918

Merged
PerfectSlayer merged 1 commit intomasterfrom
bbujon/ci-flaky-integration-vertx-redis-client
Apr 16, 2024
Merged

Mark Vert.X Redis Client 4 integration test as flaky#6918
PerfectSlayer merged 1 commit intomasterfrom
bbujon/ci-flaky-integration-vertx-redis-client

Conversation

@PerfectSlayer
Copy link
Copy Markdown
Contributor

@PerfectSlayer PerfectSlayer commented Apr 16, 2024 •
edited by atlassian Bot
Loading

What Does This Do

This PR marks Vert.X Redis Client 4 integration test as flaky.

Motivation

See #6910

Additional Notes

Jira ticket: AIDM-12

@PerfectSlayer PerfectSlayer added tag: flaky test Flaky tests inst: vertx Eclipse Vert.x instrumentation tag: flaky test/disabled Disabled flaky tests labels Apr 16, 2024
@PerfectSlayer PerfectSlayer requested a review from amarziali April 16, 2024 08:05
@PerfectSlayer PerfectSlayer requested a review from a team as a code owner April 16, 2024 08:05
@PerfectSlayer PerfectSlayer added the tag: no release notes Changes to exclude from release notes label Apr 16, 2024
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 16, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master bbujon/ci-flaky-integration-vertx-redis-client
git_commit_date 1713248465 1713254609
git_commit_sha 4c97fc1 bcc3847
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~bcc3847474
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713257978 1713257978
ci_job_id 487474075 487474075
ci_pipeline_id 32170840 32170840
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 14 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1074039
Total [baseline] (8.578 s) : 0, 8578096
Agent [candidate] (1.09 s) : 0, 1089684
Total [candidate] (8.569 s) : 0, 8569251
section iast
Agent [baseline] (1.198 s) : 0, 1198442
Total [baseline] (9.018 s) : 0, 9017994
Agent [candidate] (1.206 s) : 0, 1206163
Total [candidate] (9.02 s) : 0, 9020005
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.197 s) : 0, 1197035
Total [baseline] (8.97 s) : 0, 8969634
Agent [candidate] (1.199 s) : 0, 1199350
Total [candidate] (8.999 s) : 0, 8999415
section iast_TELEMETRY_OFF
Agent [baseline] (1.194 s) : 0, 1194264
Total [baseline] (9.032 s) : 0, 9032128
Agent [candidate] (1.198 s) : 0, 1197720
Total [candidate] (9.017 s) : 0, 9017408
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent iast 1.198 s 124.403 ms (11.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.197 s 122.996 ms (11.5%)
Agent iast_TELEMETRY_OFF 1.194 s 120.225 ms (11.2%)
Total tracing 8.578 s -
Total iast 9.018 s 439.898 ms (5.1%)
Total iast_HARDCODED_SECRET_DISABLED 8.97 s 391.539 ms (4.6%)
Total iast_TELEMETRY_OFF 9.032 s 454.032 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent iast 1.206 s 116.479 ms (10.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.199 s 109.666 ms (10.1%)
Agent iast_TELEMETRY_OFF 1.198 s 108.036 ms (9.9%)
Total tracing 8.569 s -
Total iast 9.02 s 450.754 ms (5.3%)
Total iast_HARDCODED_SECRET_DISABLED 8.999 s 430.164 ms (5.0%)
Total iast_TELEMETRY_OFF 9.017 s 448.157 ms (5.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (672.979 ms) : 0, 672979
BytebuddyAgent [candidate] (682.771 ms) : 0, 682771
GlobalTracer [baseline] (308.931 ms) : 0, 308931
GlobalTracer [candidate] (313.799 ms) : 0, 313799
AppSec [baseline] (49.476 ms) : 0, 49476
AppSec [candidate] (49.855 ms) : 0, 49855
Remote Config [baseline] (670.429 µs) : 0, 670
Remote Config [candidate] (658.805 µs) : 0, 659
Telemetry [baseline] (7.666 ms) : 0, 7666
Telemetry [candidate] (7.635 ms) : 0, 7635
section iast
BytebuddyAgent [baseline] (794.827 ms) : 0, 794827
BytebuddyAgent [candidate] (799.678 ms) : 0, 799678
GlobalTracer [baseline] (287.897 ms) : 0, 287897
GlobalTracer [candidate] (289.738 ms) : 0, 289738
AppSec [baseline] (50.806 ms) : 0, 50806
AppSec [candidate] (50.29 ms) : 0, 50290
IAST [baseline] (22.579 ms) : 0, 22579
IAST [candidate] (24.708 ms) : 0, 24708
Remote Config [baseline] (586.629 µs) : 0, 587
Remote Config [candidate] (582.372 µs) : 0, 582
Telemetry [baseline] (7.294 ms) : 0, 7294
Telemetry [candidate] (6.592 ms) : 0, 6592
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (793.68 ms) : 0, 793680
BytebuddyAgent [candidate] (793.981 ms) : 0, 793981
GlobalTracer [baseline] (287.494 ms) : 0, 287494
GlobalTracer [candidate] (288.845 ms) : 0, 288845
AppSec [baseline] (49.081 ms) : 0, 49081
AppSec [candidate] (48.73 ms) : 0, 48730
IAST [baseline] (24.418 ms) : 0, 24418
IAST [candidate] (26.114 ms) : 0, 26114
Remote Config [baseline] (577.014 µs) : 0, 577
Remote Config [candidate] (601.475 µs) : 0, 601
Telemetry [baseline] (7.38 ms) : 0, 7380
Telemetry [candidate] (6.65 ms) : 0, 6650
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (791.744 ms) : 0, 791744
BytebuddyAgent [candidate] (792.301 ms) : 0, 792301
GlobalTracer [baseline] (287.632 ms) : 0, 287632
GlobalTracer [candidate] (289.657 ms) : 0, 289657
AppSec [baseline] (50.308 ms) : 0, 50308
AppSec [candidate] (51.845 ms) : 0, 51845
IAST [baseline] (21.597 ms) : 0, 21597
IAST [candidate] (21.461 ms) : 0, 21461
Remote Config [baseline] (574.407 µs) : 0, 574
Remote Config [candidate] (584.324 µs) : 0, 584
Telemetry [baseline] (8.078 ms) : 0, 8078
Telemetry [candidate] (7.416 ms) : 0, 7416
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1076250
Total [baseline] (10.375 s) : 0, 10374801
Agent [candidate] (1.075 s) : 0, 1075086
Total [candidate] (10.376 s) : 0, 10375557
section appsec
Agent [baseline] (1.205 s) : 0, 1205230
Total [baseline] (10.443 s) : 0, 10443035
Agent [candidate] (1.197 s) : 0, 1196590
Total [candidate] (10.494 s) : 0, 10493656
section iast
Agent [baseline] (1.215 s) : 0, 1214684
Total [baseline] (10.775 s) : 0, 10775379
Agent [candidate] (1.199 s) : 0, 1199122
Total [candidate] (10.787 s) : 0, 10786886
section profiling
Agent [baseline] (1.268 s) : 0, 1267725
Total [baseline] (10.544 s) : 0, 10543689
Agent [candidate] (1.269 s) : 0, 1269225
Total [candidate] (10.602 s) : 0, 10601744
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.076 s -
Agent appsec 1.205 s 128.98 ms (12.0%)
Agent iast 1.215 s 138.434 ms (12.9%)
Agent profiling 1.268 s 191.475 ms (17.8%)
Total tracing 10.375 s -
Total appsec 10.443 s 68.234 ms (0.7%)
Total iast 10.775 s 400.578 ms (3.9%)
Total profiling 10.544 s 168.888 ms (1.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent appsec 1.197 s 121.504 ms (11.3%)
Agent iast 1.199 s 124.037 ms (11.5%)
Agent profiling 1.269 s 194.139 ms (18.1%)
Total tracing 10.376 s -
Total appsec 10.494 s 118.099 ms (1.1%)
Total iast 10.787 s 411.329 ms (4.0%)
Total profiling 10.602 s 226.187 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.182 ms) : 0, 674182
BytebuddyAgent [candidate] (673.081 ms) : 0, 673081
GlobalTracer [baseline] (309.916 ms) : 0, 309916
GlobalTracer [candidate] (309.941 ms) : 0, 309941
AppSec [baseline] (49.575 ms) : 0, 49575
AppSec [candidate] (49.353 ms) : 0, 49353
Remote Config [baseline] (661.96 µs) : 0, 662
Remote Config [candidate] (656.594 µs) : 0, 657
Telemetry [baseline] (7.613 ms) : 0, 7613
Telemetry [candidate] (7.584 ms) : 0, 7584
section appsec
BytebuddyAgent [baseline] (700.309 ms) : 0, 700309
BytebuddyAgent [candidate] (694.255 ms) : 0, 694255
GlobalTracer [baseline] (293.148 ms) : 0, 293148
GlobalTracer [candidate] (291.189 ms) : 0, 291189
AppSec [baseline] (149.863 ms) : 0, 149863
AppSec [candidate] (149.3 ms) : 0, 149300
IAST [baseline] (19.025 ms) : 0, 19025
IAST [candidate] (18.822 ms) : 0, 18822
Remote Config [baseline] (617.0 µs) : 0, 617
Remote Config [candidate] (605.554 µs) : 0, 606
Telemetry [baseline] (7.517 ms) : 0, 7517
Telemetry [candidate] (8.142 ms) : 0, 8142
section iast
BytebuddyAgent [baseline] (805.693 ms) : 0, 805693
BytebuddyAgent [candidate] (793.733 ms) : 0, 793733
GlobalTracer [baseline] (291.531 ms) : 0, 291531
GlobalTracer [candidate] (288.403 ms) : 0, 288403
AppSec [baseline] (49.103 ms) : 0, 49103
AppSec [candidate] (49.014 ms) : 0, 49014
IAST [baseline] (26.277 ms) : 0, 26277
IAST [candidate] (24.778 ms) : 0, 24778
Remote Config [baseline] (573.72 µs) : 0, 574
Remote Config [candidate] (595.96 µs) : 0, 596
Telemetry [baseline] (6.612 ms) : 0, 6612
Telemetry [candidate] (8.132 ms) : 0, 8132
section profiling
BytebuddyAgent [baseline] (677.306 ms) : 0, 677306
BytebuddyAgent [candidate] (677.16 ms) : 0, 677160
GlobalTracer [baseline] (379.967 ms) : 0, 379967
GlobalTracer [candidate] (381.316 ms) : 0, 381316
AppSec [baseline] (50.182 ms) : 0, 50182
AppSec [candidate] (50.456 ms) : 0, 50456
Remote Config [baseline] (726.51 µs) : 0, 727
Remote Config [candidate] (719.183 µs) : 0, 719
Telemetry [baseline] (7.443 ms) : 0, 7443
Telemetry [candidate] (7.442 ms) : 0, 7442
ProfilingAgent [baseline] (95.883 ms) : 0, 95883
ProfilingAgent [candidate] (95.902 ms) : 0, 95902
Profiling [baseline] (95.907 ms) : 0, 95907
Profiling [candidate] (95.926 ms) : 0, 95926
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-16T08:31:55 2024-04-16T08:53:53
git_branch master bbujon/ci-flaky-integration-vertx-redis-client
git_commit_date 1713248465 1713254609
git_commit_sha 4c97fc1 bcc3847
release_version 1.33.0-SNAPSHOT~4c97fc1a28 1.33.0-SNAPSHOT~bcc3847474
start_time 2024-04-16T08:31:43 2024-04-16T08:53:40
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713257978 1713257978
ci_job_id 487474075 487474075
ci_pipeline_id 32170840 32170840
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.48 µs) : 344, 387
.   : milestone, 365,
iast (468.384 µs) : 448, 489
.   : milestone, 468,
iast_FULL (538.56 µs) : 518, 559
.   : milestone, 539,
iast_GLOBAL (494.55 µs) : 473, 516
.   : milestone, 495,
iast_HARDCODED_SECRET_DISABLED (476.034 µs) : 455, 497
.   : milestone, 476,
iast_INACTIVE (450.607 µs) : 429, 472
.   : milestone, 451,
iast_TELEMETRY_OFF (474.293 µs) : 453, 495
.   : milestone, 474,
tracing (445.217 µs) : 425, 466
.   : milestone, 445,
section candidate
no_agent (362.462 µs) : 343, 382
.   : milestone, 362,
iast (478.531 µs) : 457, 500
.   : milestone, 479,
iast_FULL (538.515 µs) : 517, 560
.   : milestone, 539,
iast_GLOBAL (501.544 µs) : 479, 524
.   : milestone, 502,
iast_HARDCODED_SECRET_DISABLED (475.526 µs) : 454, 497
.   : milestone, 476,
iast_INACTIVE (450.634 µs) : 429, 472
.   : milestone, 451,
iast_TELEMETRY_OFF (468.45 µs) : 448, 489
.   : milestone, 468,
tracing (443.748 µs) : 423, 464
.   : milestone, 444,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.48 µs [344.267 µs, 386.694 µs] -
iast 468.384 µs [447.736 µs, 489.032 µs] 102.904 µs (28.2%)
iast_FULL 538.56 µs [517.634 µs, 559.486 µs] 173.08 µs (47.4%)
iast_GLOBAL 494.55 µs [472.913 µs, 516.187 µs] 129.07 µs (35.3%)
iast_HARDCODED_SECRET_DISABLED 476.034 µs [455.396 µs, 496.671 µs] 110.553 µs (30.2%)
iast_INACTIVE 450.607 µs [429.331 µs, 471.883 µs] 85.127 µs (23.3%)
iast_TELEMETRY_OFF 474.293 µs [453.478 µs, 495.108 µs] 108.813 µs (29.8%)
tracing 445.217 µs [424.606 µs, 465.827 µs] 79.736 µs (21.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.462 µs [342.761 µs, 382.162 µs] -
iast 478.531 µs [456.886 µs, 500.175 µs] 116.069 µs (32.0%)
iast_FULL 538.515 µs [516.632 µs, 560.398 µs] 176.053 µs (48.6%)
iast_GLOBAL 501.544 µs [478.834 µs, 524.254 µs] 139.082 µs (38.4%)
iast_HARDCODED_SECRET_DISABLED 475.526 µs [453.843 µs, 497.209 µs] 113.064 µs (31.2%)
iast_INACTIVE 450.634 µs [429.035 µs, 472.234 µs] 88.172 µs (24.3%)
iast_TELEMETRY_OFF 468.45 µs [447.898 µs, 489.003 µs] 105.989 µs (29.2%)
tracing 443.748 µs [423.062 µs, 464.433 µs] 81.286 µs (22.4%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~bcc3847474, baseline=1.33.0-SNAPSHOT~4c97fc1a28
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.337 ms) : 1318, 1356
.   : milestone, 1337,
appsec (1.718 ms) : 1693, 1742
.   : milestone, 1718,
appsec_no_iast (1.72 ms) : 1696, 1744
.   : milestone, 1720,
iast (1.47 ms) : 1448, 1492
.   : milestone, 1470,
profiling (1.497 ms) : 1473, 1522
.   : milestone, 1497,
tracing (1.465 ms) : 1441, 1490
.   : milestone, 1465,
section candidate
no_agent (1.362 ms) : 1343, 1380
.   : milestone, 1362,
appsec (1.727 ms) : 1703, 1752
.   : milestone, 1727,
appsec_no_iast (1.711 ms) : 1686, 1735
.   : milestone, 1711,
iast (1.506 ms) : 1482, 1529
.   : milestone, 1506,
profiling (1.541 ms) : 1515, 1567
.   : milestone, 1541,
tracing (1.468 ms) : 1443, 1493
.   : milestone, 1468,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.337 ms [1.318 ms, 1.356 ms] -
appsec 1.718 ms [1.693 ms, 1.742 ms] 380.214 µs (28.4%)
appsec_no_iast 1.72 ms [1.696 ms, 1.744 ms] 382.444 µs (28.6%)
iast 1.47 ms [1.448 ms, 1.492 ms] 132.602 µs (9.9%)
profiling 1.497 ms [1.473 ms, 1.522 ms] 160.096 µs (12.0%)
tracing 1.465 ms [1.441 ms, 1.49 ms] 128.027 µs (9.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.362 ms [1.343 ms, 1.38 ms] -
appsec 1.727 ms [1.703 ms, 1.752 ms] 365.688 µs (26.9%)
appsec_no_iast 1.711 ms [1.686 ms, 1.735 ms] 349.003 µs (25.6%)
iast 1.506 ms [1.482 ms, 1.529 ms] 144.392 µs (10.6%)
profiling 1.541 ms [1.515 ms, 1.567 ms] 179.927 µs (13.2%)
tracing 1.468 ms [1.443 ms, 1.493 ms] 106.396 µs (7.8%)

@PerfectSlayer PerfectSlayer merged commit 733e306 into master Apr 16, 2024
@PerfectSlayer PerfectSlayer deleted the bbujon/ci-flaky-integration-vertx-redis-client branch April 16, 2024 13:58
@github-actions github-actions Bot added this to the 1.33.0 milestone Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dougqh dougqh dougqh approved these changes

@amarziali amarziali amarziali approved these changes

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal was automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

inst: vertx Eclipse Vert.x instrumentation tag: flaky test/disabled Disabled flaky tests tag: flaky test Flaky tests tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.33.0

Development

Successfully merging this pull request may close these issues.

3 participants

@PerfectSlayer @dougqh @amarziali