Skip to content

Implement telemetry for CI Visibility APIs#6667

Merged
nikita-tkachenko-datadog merged 2 commits intomasterfrom
nikita-tkachenko/ci-vis-api-telemetry
Feb 16, 2024
Merged

Implement telemetry for CI Visibility APIs#6667
nikita-tkachenko-datadog merged 2 commits intomasterfrom
nikita-tkachenko/ci-vis-api-telemetry

Conversation

@nikita-tkachenko-datadog
Copy link
Copy Markdown
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Feb 12, 2024 •
edited by atlassian Bot
Loading

What Does This Do

Implements sending telemetry metrics for API endpoints that are specific to CI Visibility (events and coverage data intakes, Git data API, etc).

Motivation

Monitoring CI Visibility behaviour to detect unexpected errors/problems.

Jira ticket: CIVIS-2427

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Feb 12, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/coverage-telemetry branch 2 times, most recently from 2c356ba to 8ffa08c Compare February 12, 2024 14:49
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch from 9a6f79f to d201021 Compare February 12, 2024 15:33
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/coverage-telemetry branch from e106fed to 8da5d74 Compare February 13, 2024 19:22
Base automatically changed from nikita-tkachenko/coverage-telemetry to master February 14, 2024 15:03
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch 2 times, most recently from de7d9a3 to 5de553e Compare February 14, 2024 21:08
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Feb 14, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/ci-vis-api-telemetry
git_commit_date 1708008561 1708008633
git_commit_sha 142c8ca e24476c
release_version 1.31.0-SNAPSHOT~142c8cac77 1.31.0-SNAPSHOT~e24476c597
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1708011424 1708011424
ci_job_id 435108661 435108661
ci_pipeline_id 28356874 28356874
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.31.0-SNAPSHOT~e24476c597, baseline=1.31.0-SNAPSHOT~142c8cac77

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.062 s) : 0, 1062318
Total [baseline] (9.396 s) : 0, 9395504
Agent [candidate] (1.061 s) : 0, 1060511
Total [candidate] (9.349 s) : 0, 9348779
section appsec
Agent [baseline] (1.155 s) : 0, 1155453
Total [baseline] (9.424 s) : 0, 9424165
Agent [candidate] (1.16 s) : 0, 1159720
Total [candidate] (9.456 s) : 0, 9455896
section iast
Agent [baseline] (1.182 s) : 0, 1181744
Total [baseline] (9.64 s) : 0, 9640239
Agent [candidate] (1.184 s) : 0, 1183649
Total [candidate] (9.752 s) : 0, 9751955
section profiling
Agent [baseline] (1.277 s) : 0, 1276676
Total [baseline] (9.598 s) : 0, 9597906
Agent [candidate] (1.278 s) : 0, 1277957
Total [candidate] (9.578 s) : 0, 9577658
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent appsec 1.155 s 93.135 ms (8.8%)
Agent iast 1.182 s 119.426 ms (11.2%)
Agent profiling 1.277 s 214.358 ms (20.2%)
Total tracing 9.396 s -
Total appsec 9.424 s 28.661 ms (0.3%)
Total iast 9.64 s 244.735 ms (2.6%)
Total profiling 9.598 s 202.402 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent appsec 1.16 s 99.209 ms (9.4%)
Agent iast 1.184 s 123.138 ms (11.6%)
Agent profiling 1.278 s 217.446 ms (20.5%)
Total tracing 9.349 s -
Total appsec 9.456 s 107.116 ms (1.1%)
Total iast 9.752 s 403.176 ms (4.3%)
Total profiling 9.578 s 228.878 ms (2.4%) 
gantt
    title petclinic - break down per module: candidate=1.31.0-SNAPSHOT~e24476c597, baseline=1.31.0-SNAPSHOT~142c8cac77

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (668.828 ms) : 0, 668828
BytebuddyAgent [candidate] (667.346 ms) : 0, 667346
GlobalTracer [baseline] (298.85 ms) : 0, 298850
GlobalTracer [candidate] (298.774 ms) : 0, 298774
AppSec [baseline] (51.933 ms) : 0, 51933
AppSec [candidate] (51.844 ms) : 0, 51844
Remote Config [baseline] (696.744 µs) : 0, 697
Remote Config [candidate] (707.57 µs) : 0, 708
Telemetry [baseline] (7.577 ms) : 0, 7577
Telemetry [candidate] (7.651 ms) : 0, 7651
section appsec
BytebuddyAgent [baseline] (665.45 ms) : 0, 665450
BytebuddyAgent [candidate] (668.429 ms) : 0, 668429
GlobalTracer [baseline] (297.758 ms) : 0, 297758
GlobalTracer [candidate] (299.035 ms) : 0, 299035
AppSec [baseline] (150.539 ms) : 0, 150539
AppSec [candidate] (150.584 ms) : 0, 150584
Remote Config [baseline] (635.71 µs) : 0, 636
Remote Config [candidate] (638.73 µs) : 0, 639
Telemetry [baseline] (6.832 ms) : 0, 6832
Telemetry [candidate] (6.813 ms) : 0, 6813
section iast
BytebuddyAgent [baseline] (777.525 ms) : 0, 777525
BytebuddyAgent [candidate] (777.718 ms) : 0, 777718
GlobalTracer [baseline] (289.166 ms) : 0, 289166
GlobalTracer [candidate] (289.487 ms) : 0, 289487
AppSec [baseline] (53.721 ms) : 0, 53721
AppSec [candidate] (51.869 ms) : 0, 51869
IAST [baseline] (19.836 ms) : 0, 19836
IAST [candidate] (23.017 ms) : 0, 23017
Remote Config [baseline] (621.742 µs) : 0, 622
Remote Config [candidate] (693.145 µs) : 0, 693
Telemetry [baseline] (6.671 ms) : 0, 6671
Telemetry [candidate] (6.568 ms) : 0, 6568
section profiling
BytebuddyAgent [baseline] (664.967 ms) : 0, 664967
BytebuddyAgent [candidate] (665.705 ms) : 0, 665705
GlobalTracer [baseline] (381.853 ms) : 0, 381853
GlobalTracer [candidate] (382.443 ms) : 0, 382443
AppSec [baseline] (52.138 ms) : 0, 52138
AppSec [candidate] (52.003 ms) : 0, 52003
Remote Config [baseline] (658.536 µs) : 0, 659
Remote Config [candidate] (657.515 µs) : 0, 658
Telemetry [baseline] (10.92 ms) : 0, 10920
Telemetry [candidate] (12.914 ms) : 0, 12914
ProfilingAgent [baseline] (111.369 ms) : 0, 111369
ProfilingAgent [candidate] (109.603 ms) : 0, 109603
Profiling [baseline] (111.394 ms) : 0, 111394
Profiling [candidate] (109.628 ms) : 0, 109628
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-02-15T15:13:08 2024-02-15T15:32:02
git_branch master nikita-tkachenko/ci-vis-api-telemetry
git_commit_date 1708008561 1708008633
git_commit_sha 142c8ca e24476c
release_version 1.31.0-SNAPSHOT~142c8cac77 1.31.0-SNAPSHOT~e24476c597
start_time 2024-02-15T15:12:55 2024-02-15T15:31:49
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1708011424 1708011424
ci_job_id 435108661 435108661
ci_pipeline_id 28356874 28356874
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~e24476c597, baseline=1.31.0-SNAPSHOT~142c8cac77
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.352 ms) : 1333, 1371
.   : milestone, 1352,
appsec (1.768 ms) : 1743, 1793
.   : milestone, 1768,
iast (1.481 ms) : 1456, 1505
.   : milestone, 1481,
profiling (1.573 ms) : 1547, 1599
.   : milestone, 1573,
tracing (1.495 ms) : 1470, 1520
.   : milestone, 1495,
section candidate
no_agent (1.355 ms) : 1336, 1374
.   : milestone, 1355,
appsec (1.782 ms) : 1756, 1808
.   : milestone, 1782,
iast (1.502 ms) : 1477, 1526
.   : milestone, 1502,
profiling (1.555 ms) : 1529, 1581
.   : milestone, 1555,
tracing (1.504 ms) : 1479, 1529
.   : milestone, 1504,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.352 ms [1.333 ms, 1.371 ms] -
appsec 1.768 ms [1.743 ms, 1.793 ms] 416.334 µs (30.8%)
iast 1.481 ms [1.456 ms, 1.505 ms] 128.809 µs (9.5%)
profiling 1.573 ms [1.547 ms, 1.599 ms] 221.172 µs (16.4%)
tracing 1.495 ms [1.47 ms, 1.52 ms] 143.001 µs (10.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.355 ms [1.336 ms, 1.374 ms] -
appsec 1.782 ms [1.756 ms, 1.808 ms] 426.231 µs (31.4%)
iast 1.502 ms [1.477 ms, 1.526 ms] 146.476 µs (10.8%)
profiling 1.555 ms [1.529 ms, 1.581 ms] 199.446 µs (14.7%)
tracing 1.504 ms [1.479 ms, 1.529 ms] 148.6 µs (11.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~e24476c597, baseline=1.31.0-SNAPSHOT~142c8cac77
    dateFormat X
    axisFormat %s
section baseline
no_agent (369.755 µs) : 349, 390
.   : milestone, 370,
iast (468.84 µs) : 448, 490
.   : milestone, 469,
iast_FULL (533.422 µs) : 512, 554
.   : milestone, 533,
iast_GLOBAL (485.281 µs) : 464, 506
.   : milestone, 485,
iast_HARDCODED_SECRET_DISABLED (473.502 µs) : 452, 495
.   : milestone, 474,
iast_INACTIVE (444.057 µs) : 423, 465
.   : milestone, 444,
iast_TELEMETRY_OFF (477.641 µs) : 457, 499
.   : milestone, 478,
tracing (439.705 µs) : 419, 460
.   : milestone, 440,
section candidate
no_agent (368.033 µs) : 348, 388
.   : milestone, 368,
iast (469.171 µs) : 448, 490
.   : milestone, 469,
iast_FULL (534.892 µs) : 514, 556
.   : milestone, 535,
iast_GLOBAL (484.708 µs) : 464, 505
.   : milestone, 485,
iast_HARDCODED_SECRET_DISABLED (469.054 µs) : 448, 490
.   : milestone, 469,
iast_INACTIVE (440.982 µs) : 420, 462
.   : milestone, 441,
iast_TELEMETRY_OFF (467.97 µs) : 447, 489
.   : milestone, 468,
tracing (445.507 µs) : 424, 467
.   : milestone, 446,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 369.755 µs [349.344 µs, 390.165 µs] -
iast 468.84 µs [448.105 µs, 489.576 µs] 99.086 µs (26.8%)
iast_FULL 533.422 µs [512.429 µs, 554.415 µs] 163.668 µs (44.3%)
iast_GLOBAL 485.281 µs [464.4 µs, 506.163 µs] 115.527 µs (31.2%)
iast_HARDCODED_SECRET_DISABLED 473.502 µs [452.285 µs, 494.719 µs] 103.748 µs (28.1%)
iast_INACTIVE 444.057 µs [423.19 µs, 464.924 µs] 74.303 µs (20.1%)
iast_TELEMETRY_OFF 477.641 µs [456.735 µs, 498.547 µs] 107.886 µs (29.2%)
tracing 439.705 µs [419.112 µs, 460.297 µs] 69.95 µs (18.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.033 µs [348.217 µs, 387.849 µs] -
iast 469.171 µs [448.413 µs, 489.929 µs] 101.138 µs (27.5%)
iast_FULL 534.892 µs [514.168 µs, 555.616 µs] 166.859 µs (45.3%)
iast_GLOBAL 484.708 µs [464.446 µs, 504.969 µs] 116.675 µs (31.7%)
iast_HARDCODED_SECRET_DISABLED 469.054 µs [447.919 µs, 490.188 µs] 101.02 µs (27.4%)
iast_INACTIVE 440.982 µs [420.379 µs, 461.586 µs] 72.949 µs (19.8%)
iast_TELEMETRY_OFF 467.97 µs [447.162 µs, 488.778 µs] 99.937 µs (27.2%)
tracing 445.507 µs [424.444 µs, 466.57 µs] 77.474 µs (21.1%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch from 5de553e to 6b758bf Compare February 15, 2024 09:48
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review February 15, 2024 14:46
@nikita-tkachenko-datadog nikita-tkachenko-datadog requested a review from a team as a code owner February 15, 2024 14:46
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit b550bc8 into master Feb 16, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/ci-vis-api-telemetry branch February 16, 2024 10:05
@github-actions github-actions Bot added this to the 1.31.0 milestone Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@juan-fernandez juan-fernandez juan-fernandez approved these changes

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility

Projects

None yet

Milestone

1.31.0

Development

Successfully merging this pull request may close these issues.

2 participants

@nikita-tkachenko-datadog @juan-fernandez