Skip to content

Implement early flakiness detection#6727

Merged
nikita-tkachenko-datadog merged 5 commits into
masterfrom
nikita-tkachenko/early-flakiness-detection
Feb 27, 2024
Merged

Implement early flakiness detection#6727
nikita-tkachenko-datadog merged 5 commits into
masterfrom
nikita-tkachenko/early-flakiness-detection

Conversation

@nikita-tkachenko-datadog

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Feb 23, 2024 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Implements early flakiness detection CI Visibility feature (RFC).

Jira ticket: CIVIS-8329

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Feb 23, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/early-flakiness-detection branch 3 times, most recently from ca8438f to a7b55a4 Compare February 23, 2024 16:17
@pr-commenter

pr-commenter Bot commented Feb 23, 2024 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/early-flakiness-detection
git_commit_date 1709036372 1709037393
git_commit_sha ef26b91 4d273d4
release_version 1.31.0-SNAPSHOT~ef26b91da3 1.31.0-SNAPSHOT~4d273d473c
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1709040230 1709040230
ci_job_id 444375749 444375749
ci_pipeline_id 29011801 29011801
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 42 metrics, 12 unstable metrics.

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-02-27T13:00:16 2024-02-27T13:18:51
git_branch master nikita-tkachenko/early-flakiness-detection
git_commit_date 1709036372 1709037393
git_commit_sha ef26b91 4d273d4
release_version 1.31.0-SNAPSHOT~ef26b91da3 1.31.0-SNAPSHOT~4d273d473c
start_time 2024-02-27T13:00:03 2024-02-27T13:18:39
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1709040230 1709040230
ci_job_id 444375749 444375749
ci_pipeline_id 29011801 29011801
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 16 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling better
[-100.100µs; -47.672µs] or [-6.296%; -2.998%]		 unstable
[-331.017op/s; +770.577op/s] or [-11.586%; +26.970%]		 1.516ms 3076.923op/s 1.590ms 2857.143op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~4d273d473c, baseline=1.31.0-SNAPSHOT~ef26b91da3
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.578 µs) : 350, 395
.   : milestone, 373,
iast (474.758 µs) : 454, 495
.   : milestone, 475,
iast_FULL (541.095 µs) : 521, 562
.   : milestone, 541,
iast_GLOBAL (498.614 µs) : 477, 520
.   : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (478.317 µs) : 458, 499
.   : milestone, 478,
iast_INACTIVE (444.877 µs) : 424, 465
.   : milestone, 445,
iast_TELEMETRY_OFF (473.438 µs) : 453, 494
.   : milestone, 473,
tracing (436.894 µs) : 416, 457
.   : milestone, 437,
section candidate
no_agent (363.348 µs) : 343, 384
.   : milestone, 363,
iast (470.209 µs) : 450, 491
.   : milestone, 470,
iast_FULL (538.962 µs) : 519, 559
.   : milestone, 539,
iast_GLOBAL (489.187 µs) : 469, 510
.   : milestone, 489,
iast_HARDCODED_SECRET_DISABLED (472.981 µs) : 452, 494
.   : milestone, 473,
iast_INACTIVE (445.361 µs) : 424, 467
.   : milestone, 445,
iast_TELEMETRY_OFF (469.753 µs) : 449, 490
.   : milestone, 470,
tracing (439.042 µs) : 419, 459
.   : milestone, 439,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 372.578 µs [350.086 µs, 395.069 µs] -
iast 474.758 µs [454.229 µs, 495.287 µs] 102.181 µs (27.4%)
iast_FULL 541.095 µs [520.518 µs, 561.673 µs] 168.518 µs (45.2%)
iast_GLOBAL 498.614 µs [477.22 µs, 520.009 µs] 126.037 µs (33.8%)
iast_HARDCODED_SECRET_DISABLED 478.317 µs [457.8 µs, 498.834 µs] 105.739 µs (28.4%)
iast_INACTIVE 444.877 µs [424.445 µs, 465.309 µs] 72.299 µs (19.4%)
iast_TELEMETRY_OFF 473.438 µs [452.704 µs, 494.173 µs] 100.861 µs (27.1%)
tracing 436.894 µs [416.335 µs, 457.453 µs] 64.317 µs (17.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.348 µs [343.026 µs, 383.669 µs] -
iast 470.209 µs [449.669 µs, 490.75 µs] 106.862 µs (29.4%)
iast_FULL 538.962 µs [518.58 µs, 559.344 µs] 175.615 µs (48.3%)
iast_GLOBAL 489.187 µs [468.86 µs, 509.515 µs] 125.84 µs (34.6%)
iast_HARDCODED_SECRET_DISABLED 472.981 µs [452.014 µs, 493.949 µs] 109.633 µs (30.2%)
iast_INACTIVE 445.361 µs [424.121 µs, 466.601 µs] 82.014 µs (22.6%)
iast_TELEMETRY_OFF 469.753 µs [449.085 µs, 490.42 µs] 106.405 µs (29.3%)
tracing 439.042 µs [418.94 µs, 459.145 µs] 75.695 µs (20.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~4d273d473c, baseline=1.31.0-SNAPSHOT~ef26b91da3
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.344 ms) : 1325, 1363
.   : milestone, 1344,
appsec (1.762 ms) : 1738, 1785
.   : milestone, 1762,
iast (1.509 ms) : 1486, 1532
.   : milestone, 1509,
profiling (1.59 ms) : 1565, 1615
.   : milestone, 1590,
tracing (1.495 ms) : 1472, 1518
.   : milestone, 1495,
section candidate
no_agent (1.347 ms) : 1328, 1366
.   : milestone, 1347,
appsec (1.787 ms) : 1763, 1810
.   : milestone, 1787,
iast (1.513 ms) : 1489, 1536
.   : milestone, 1513,
profiling (1.516 ms) : 1492, 1540
.   : milestone, 1516,
tracing (1.496 ms) : 1472, 1519
.   : milestone, 1496,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.344 ms [1.325 ms, 1.363 ms] -
appsec 1.762 ms [1.738 ms, 1.785 ms] 417.682 µs (31.1%)
iast 1.509 ms [1.486 ms, 1.532 ms] 164.486 µs (12.2%)
profiling 1.59 ms [1.565 ms, 1.615 ms] 245.778 µs (18.3%)
tracing 1.495 ms [1.472 ms, 1.518 ms] 150.53 µs (11.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.347 ms [1.328 ms, 1.366 ms] -
appsec 1.787 ms [1.763 ms, 1.81 ms] 439.599 µs (32.6%)
iast 1.513 ms [1.489 ms, 1.536 ms] 165.827 µs (12.3%)
profiling 1.516 ms [1.492 ms, 1.54 ms] 169.107 µs (12.6%)
tracing 1.496 ms [1.472 ms, 1.519 ms] 148.775 µs (11.0%)

Base automatically changed from nikita-tkachenko/ci-visibility-itr-correlation-id to master February 23, 2024 20:39
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/early-flakiness-detection branch from a7b55a4 to b1243c5 Compare February 23, 2024 20:40
@nikita-tkachenko-datadog
nikita-tkachenko-datadog marked this pull request as ready for review February 27, 2024 09:19
@nikita-tkachenko-datadog
nikita-tkachenko-datadog merged commit 2f0211b into master Feb 27, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog deleted the nikita-tkachenko/early-flakiness-detection branch February 27, 2024 14:12
@github-actions github-actions Bot added this to the 1.31.0 milestone Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes
@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java
@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility

Projects

None yet

Milestone

1.31.0

Development

Successfully merging this pull request may close these issues.

2 participants

@nikita-tkachenko-datadog @anmarchenko