Skip to content

Implement telemetry for CI Visibility intakes#6668

Merged
nikita-tkachenko-datadog merged 1 commit intomasterfrom
nikita-tkachenko/ci-vis-intake-telemetry
Feb 16, 2024
Merged

Implement telemetry for CI Visibility intakes#6668
nikita-tkachenko-datadog merged 1 commit intomasterfrom
nikita-tkachenko/ci-vis-intake-telemetry

Conversation

@nikita-tkachenko-datadog
Copy link
Copy Markdown
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Feb 12, 2024 •
edited by atlassian Bot
Loading

What Does This Do

Implements sending telemetry metrics for CI Visibility intakes (test events intake and per-test code coverages intake).

Motivation

Monitoring CI Visibility behaviour to detect unexpected errors/problems.

Jira ticket: CIVIS-2427

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Feb 12, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch 2 times, most recently from d201021 to de7d9a3 Compare February 14, 2024 21:02
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-intake-telemetry branch from 8946f5a to cebc6cd Compare February 14, 2024 21:05
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch from de7d9a3 to 5de553e Compare February 14, 2024 21:08
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-intake-telemetry branch from cebc6cd to 8be95d6 Compare February 14, 2024 21:08
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Feb 14, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/ci-vis-intake-telemetry
git_commit_date 1708077936 1708077992
git_commit_sha b550bc8 f50e509
release_version 1.31.0-SNAPSHOT~b550bc8737 1.31.0-SNAPSHOT~f50e5093f7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1708080787 1708080787
ci_job_id 435909285 435909285
ci_pipeline_id 28409488 28409488
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 4 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.31.0-SNAPSHOT~f50e5093f7, baseline=1.31.0-SNAPSHOT~b550bc8737

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055771
Total [baseline] (9.129 s) : 0, 9129268
Agent [candidate] (1.056 s) : 0, 1055550
Total [candidate] (9.12 s) : 0, 9119553
section appsec
Agent [baseline] (1.162 s) : 0, 1161640
Total [baseline] (9.265 s) : 0, 9264911
Agent [candidate] (1.153 s) : 0, 1153001
Total [candidate] (9.296 s) : 0, 9296264
section iast
Agent [baseline] (1.18 s) : 0, 1179924
Total [baseline] (9.441 s) : 0, 9441019
Agent [candidate] (1.184 s) : 0, 1184330
Total [candidate] (9.439 s) : 0, 9439108
section profiling
Agent [baseline] (1.266 s) : 0, 1266247
Total [baseline] (9.328 s) : 0, 9327855
Agent [candidate] (1.269 s) : 0, 1269026
Total [candidate] (9.408 s) : 0, 9407804
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent appsec 1.162 s 105.869 ms (10.0%)
Agent iast 1.18 s 124.153 ms (11.8%)
Agent profiling 1.266 s 210.476 ms (19.9%)
Total tracing 9.129 s -
Total appsec 9.265 s 135.643 ms (1.5%)
Total iast 9.441 s 311.751 ms (3.4%)
Total profiling 9.328 s 198.587 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.056 s -
Agent appsec 1.153 s 97.45 ms (9.2%)
Agent iast 1.184 s 128.779 ms (12.2%)
Agent profiling 1.269 s 213.476 ms (20.2%)
Total tracing 9.12 s -
Total appsec 9.296 s 176.711 ms (1.9%)
Total iast 9.439 s 319.555 ms (3.5%)
Total profiling 9.408 s 288.25 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.31.0-SNAPSHOT~f50e5093f7, baseline=1.31.0-SNAPSHOT~b550bc8737

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (666.649 ms) : 0, 666649
BytebuddyAgent [candidate] (665.838 ms) : 0, 665838
GlobalTracer [baseline] (294.828 ms) : 0, 294828
GlobalTracer [candidate] (295.353 ms) : 0, 295353
AppSec [baseline] (51.527 ms) : 0, 51527
AppSec [candidate] (51.634 ms) : 0, 51634
Remote Config [baseline] (714.387 µs) : 0, 714
Remote Config [candidate] (713.418 µs) : 0, 713
Telemetry [baseline] (7.656 ms) : 0, 7656
Telemetry [candidate] (7.709 ms) : 0, 7709
section appsec
BytebuddyAgent [baseline] (671.24 ms) : 0, 671240
BytebuddyAgent [candidate] (665.597 ms) : 0, 665597
GlobalTracer [baseline] (297.195 ms) : 0, 297195
GlobalTracer [candidate] (294.798 ms) : 0, 294798
AppSec [baseline] (151.044 ms) : 0, 151044
AppSec [candidate] (150.936 ms) : 0, 150936
Remote Config [baseline] (641.3 µs) : 0, 641
Remote Config [candidate] (633.725 µs) : 0, 634
Telemetry [baseline] (6.891 ms) : 0, 6891
Telemetry [candidate] (6.797 ms) : 0, 6797
section iast
BytebuddyAgent [baseline] (778.887 ms) : 0, 778887
BytebuddyAgent [candidate] (781.615 ms) : 0, 781615
GlobalTracer [baseline] (286.495 ms) : 0, 286495
GlobalTracer [candidate] (287.932 ms) : 0, 287932
AppSec [baseline] (49.805 ms) : 0, 49805
AppSec [candidate] (49.465 ms) : 0, 49465
Remote Config [baseline] (629.917 µs) : 0, 630
Remote Config [candidate] (628.497 µs) : 0, 628
Telemetry [baseline] (6.817 ms) : 0, 6817
Telemetry [candidate] (9.093 ms) : 0, 9093
IAST [baseline] (23.012 ms) : 0, 23012
IAST [candidate] (21.149 ms) : 0, 21149
section profiling
ProfilingAgent [baseline] (115.53 ms) : 0, 115530
ProfilingAgent [candidate] (116.325 ms) : 0, 116325
BytebuddyAgent [baseline] (658.628 ms) : 0, 658628
BytebuddyAgent [candidate] (659.836 ms) : 0, 659836
GlobalTracer [baseline] (377.617 ms) : 0, 377617
GlobalTracer [candidate] (378.343 ms) : 0, 378343
AppSec [baseline] (51.791 ms) : 0, 51791
AppSec [candidate] (51.878 ms) : 0, 51878
Remote Config [baseline] (670.606 µs) : 0, 671
Remote Config [candidate] (658.908 µs) : 0, 659
Telemetry [baseline] (7.532 ms) : 0, 7532
Telemetry [candidate] (7.511 ms) : 0, 7511
Profiling [baseline] (115.554 ms) : 0, 115554
Profiling [candidate] (116.351 ms) : 0, 116351
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-02-16T10:29:35 2024-02-16T10:48:09
git_branch master nikita-tkachenko/ci-vis-intake-telemetry
git_commit_date 1708077936 1708077992
git_commit_sha b550bc8 f50e509
release_version 1.31.0-SNAPSHOT~b550bc8737 1.31.0-SNAPSHOT~f50e5093f7
start_time 2024-02-16T10:29:22 2024-02-16T10:47:56
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1708080787 1708080787
ci_job_id 435909285 435909285
ci_pipeline_id 28409488 28409488
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 15 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~f50e5093f7, baseline=1.31.0-SNAPSHOT~b550bc8737
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.76 ms) : 1736, 1785
.   : milestone, 1760,
iast (1.495 ms) : 1471, 1518
.   : milestone, 1495,
profiling (1.509 ms) : 1484, 1534
.   : milestone, 1509,
tracing (1.52 ms) : 1496, 1543
.   : milestone, 1520,
section candidate
no_agent (1.346 ms) : 1327, 1365
.   : milestone, 1346,
appsec (1.776 ms) : 1752, 1800
.   : milestone, 1776,
iast (1.522 ms) : 1498, 1546
.   : milestone, 1522,
profiling (1.52 ms) : 1496, 1544
.   : milestone, 1520,
tracing (1.508 ms) : 1485, 1532
.   : milestone, 1508,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.76 ms [1.736 ms, 1.785 ms] 418.188 µs (31.2%)
iast 1.495 ms [1.471 ms, 1.518 ms] 152.592 µs (11.4%)
profiling 1.509 ms [1.484 ms, 1.534 ms] 166.718 µs (12.4%)
tracing 1.52 ms [1.496 ms, 1.543 ms] 177.717 µs (13.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.346 ms [1.327 ms, 1.365 ms] -
appsec 1.776 ms [1.752 ms, 1.8 ms] 429.955 µs (31.9%)
iast 1.522 ms [1.498 ms, 1.546 ms] 175.796 µs (13.1%)
profiling 1.52 ms [1.496 ms, 1.544 ms] 174.127 µs (12.9%)
tracing 1.508 ms [1.485 ms, 1.532 ms] 162.223 µs (12.1%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.31.0-SNAPSHOT~f50e5093f7, baseline=1.31.0-SNAPSHOT~b550bc8737
    dateFormat X
    axisFormat %s
section baseline
no_agent (362.366 µs) : 342, 383
.   : milestone, 362,
iast (475.645 µs) : 455, 496
.   : milestone, 476,
iast_FULL (534.945 µs) : 514, 555
.   : milestone, 535,
iast_GLOBAL (488.306 µs) : 468, 508
.   : milestone, 488,
iast_HARDCODED_SECRET_DISABLED (471.209 µs) : 451, 492
.   : milestone, 471,
iast_INACTIVE (446.865 µs) : 426, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (463.306 µs) : 443, 484
.   : milestone, 463,
tracing (440.954 µs) : 420, 461
.   : milestone, 441,
section candidate
no_agent (362.674 µs) : 342, 383
.   : milestone, 363,
iast (468.208 µs) : 448, 489
.   : milestone, 468,
iast_FULL (529.909 µs) : 509, 551
.   : milestone, 530,
iast_GLOBAL (498.423 µs) : 476, 521
.   : milestone, 498,
iast_HARDCODED_SECRET_DISABLED (465.955 µs) : 446, 486
.   : milestone, 466,
iast_INACTIVE (444.42 µs) : 424, 465
.   : milestone, 444,
iast_TELEMETRY_OFF (467.594 µs) : 447, 488
.   : milestone, 468,
tracing (441.319 µs) : 421, 462
.   : milestone, 441,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.366 µs [342.107 µs, 382.625 µs] -
iast 475.645 µs [454.791 µs, 496.499 µs] 113.279 µs (31.3%)
iast_FULL 534.945 µs [514.408 µs, 555.482 µs] 172.579 µs (47.6%)
iast_GLOBAL 488.306 µs [468.178 µs, 508.434 µs] 125.94 µs (34.8%)
iast_HARDCODED_SECRET_DISABLED 471.209 µs [450.626 µs, 491.792 µs] 108.843 µs (30.0%)
iast_INACTIVE 446.865 µs [426.073 µs, 467.658 µs] 84.499 µs (23.3%)
iast_TELEMETRY_OFF 463.306 µs [442.532 µs, 484.081 µs] 100.94 µs (27.9%)
tracing 440.954 µs [420.419 µs, 461.49 µs] 78.588 µs (21.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.674 µs [342.041 µs, 383.306 µs] -
iast 468.208 µs [447.597 µs, 488.819 µs] 105.534 µs (29.1%)
iast_FULL 529.909 µs [508.871 µs, 550.947 µs] 167.235 µs (46.1%)
iast_GLOBAL 498.423 µs [476.341 µs, 520.504 µs] 135.749 µs (37.4%)
iast_HARDCODED_SECRET_DISABLED 465.955 µs [445.542 µs, 486.369 µs] 103.282 µs (28.5%)
iast_INACTIVE 444.42 µs [423.509 µs, 465.331 µs] 81.746 µs (22.5%)
iast_TELEMETRY_OFF 467.594 µs [447.235 µs, 487.954 µs] 104.921 µs (28.9%)
tracing 441.319 µs [420.899 µs, 461.739 µs] 78.645 µs (21.7%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-api-telemetry branch from 5de553e to 6b758bf Compare February 15, 2024 09:48
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-intake-telemetry branch 3 times, most recently from 3899ba0 to f06c5fd Compare February 16, 2024 09:22
Base automatically changed from nikita-tkachenko/ci-vis-api-telemetry to master February 16, 2024 10:05
@nikita-tkachenko-datadog nikita-tkachenko-datadog force-pushed the nikita-tkachenko/ci-vis-intake-telemetry branch from f06c5fd to f50e509 Compare February 16, 2024 10:06
@nikita-tkachenko-datadog nikita-tkachenko-datadog marked this pull request as ready for review February 16, 2024 13:09
@nikita-tkachenko-datadog nikita-tkachenko-datadog merged commit aee3ca5 into master Feb 16, 2024
@nikita-tkachenko-datadog nikita-tkachenko-datadog deleted the nikita-tkachenko/ci-vis-intake-telemetry branch February 16, 2024 13:20
@github-actions github-actions Bot added this to the 1.31.0 milestone Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@anmarchenko anmarchenko anmarchenko approved these changes

@PerfectSlayer PerfectSlayer Awaiting requested review from PerfectSlayer PerfectSlayer is a code owner automatically assigned from DataDog/apm-java

@ygree ygree Awaiting requested review from ygree ygree is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility

Projects

None yet

Milestone

1.31.0

Development

Successfully merging this pull request may close these issues.

2 participants

@nikita-tkachenko-datadog @anmarchenko