Skip to content

ITR code coverage support#7367

Merged
nikita-tkachenko-datadog merged 18 commits into
masterfrom
nikita-tkachenko/code-coverage-with-itr
Sep 10, 2024
Merged

ITR code coverage support#7367
nikita-tkachenko-datadog merged 18 commits into
masterfrom
nikita-tkachenko/code-coverage-with-itr

Conversation

@nikita-tkachenko-datadog

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Jul 29, 2024

Copy link
Copy Markdown
Contributor

What Does This Do

Updates the logic that calculates code coverage percentage (test.code_coverage.lines_pct tag in session and module events) to take into account tests skipped by ITR.

Motivation

Showing "correct" coverage percentage for sessions where ITR tests skipping is enabled.
Here "correct" means "as if no tests were skipped".

Additional Notes

When requesting the list of skippable tests from the backend, the tracer also receives coverage data for these tests: a bitmap of covered lines for every file covered.
The tracer uses this data to "backfill" the coverage obtained at runtime.

Contributor Checklist

Jira ticket: SDTEST-112

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Jul 29, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch from 28a956d to 812e6a2 Compare July 29, 2024 16:10
@pr-commenter

pr-commenter Bot commented Jul 29, 2024

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/code-coverage-with-itr
git_commit_date 1725890539 1725897527
git_commit_sha 1d59e61 c0c0f53
release_version 1.40.0-SNAPSHOT~1d59e612a6 1.40.0-SNAPSHOT~c0c0f5394f
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725899922 1725899922
ci_job_id 632289988 632289988
ci_pipeline_id 43877319 43877319
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 12 unstable metrics.

Startup time reports for petclinic
gantt
    title petclinic - global startup overhead: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.058 s) : 0, 1058470
Total [baseline] (10.405 s) : 0, 10404701
Agent [candidate] (1.053 s) : 0, 1053180
Total [candidate] (10.35 s) : 0, 10349854
section appsec
Agent [baseline] (1.188 s) : 0, 1188350
Total [baseline] (10.578 s) : 0, 10577993
Agent [candidate] (1.183 s) : 0, 1182948
Total [candidate] (10.599 s) : 0, 10599115
section iast
Agent [baseline] (1.186 s) : 0, 1185643
Total [baseline] (10.818 s) : 0, 10818357
Agent [candidate] (1.183 s) : 0, 1183263
Total [candidate] (10.816 s) : 0, 10815824
section profiling
Agent [baseline] (1.252 s) : 0, 1252496
Total [baseline] (10.7 s) : 0, 10700059
Agent [candidate] (1.251 s) : 0, 1250650
Total [candidate] (10.667 s) : 0, 10666993
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.058 s -
Agent appsec 1.188 s 129.88 ms (12.3%)
Agent iast 1.186 s 127.173 ms (12.0%)
Agent profiling 1.252 s 194.026 ms (18.3%)
Total tracing 10.405 s -
Total appsec 10.578 s 173.292 ms (1.7%)
Total iast 10.818 s 413.656 ms (4.0%)
Total profiling 10.7 s 295.358 ms (2.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent appsec 1.183 s 129.768 ms (12.3%)
Agent iast 1.183 s 130.083 ms (12.4%)
Agent profiling 1.251 s 197.47 ms (18.7%)
Total tracing 10.35 s -
Total appsec 10.599 s 249.261 ms (2.4%)
Total iast 10.816 s 465.97 ms (4.5%)
Total profiling 10.667 s 317.139 ms (3.1%)
gantt
    title petclinic - break down per module: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (675.53 ms) : 0, 675530
BytebuddyAgent [candidate] (671.873 ms) : 0, 671873
GlobalTracer [baseline] (309.435 ms) : 0, 309435
GlobalTracer [candidate] (308.097 ms) : 0, 308097
AppSec [baseline] (51.601 ms) : 0, 51601
AppSec [candidate] (51.358 ms) : 0, 51358
Remote Config [baseline] (679.661 µs) : 0, 680
Remote Config [candidate] (688.945 µs) : 0, 689
Telemetry [baseline] (7.551 ms) : 0, 7551
Telemetry [candidate] (7.58 ms) : 0, 7580
section appsec
BytebuddyAgent [baseline] (693.267 ms) : 0, 693267
BytebuddyAgent [candidate] (689.262 ms) : 0, 689262
GlobalTracer [baseline] (302.296 ms) : 0, 302296
GlobalTracer [candidate] (300.584 ms) : 0, 300584
AppSec [baseline] (159.4 ms) : 0, 159400
AppSec [candidate] (159.573 ms) : 0, 159573
Remote Config [baseline] (631.38 µs) : 0, 631
Remote Config [candidate] (627.009 µs) : 0, 627
Telemetry [baseline] (8.192 ms) : 0, 8192
Telemetry [candidate] (9.547 ms) : 0, 9547
IAST [baseline] (21.884 ms) : 0, 21884
IAST [candidate] (19.613 ms) : 0, 19613
section iast
BytebuddyAgent [baseline] (788.776 ms) : 0, 788776
BytebuddyAgent [candidate] (786.85 ms) : 0, 786850
GlobalTracer [baseline] (298.397 ms) : 0, 298397
GlobalTracer [candidate] (298.778 ms) : 0, 298778
AppSec [baseline] (54.825 ms) : 0, 54825
AppSec [candidate] (51.594 ms) : 0, 51594
Remote Config [baseline] (596.707 µs) : 0, 597
Remote Config [candidate] (592.817 µs) : 0, 593
Telemetry [baseline] (7.425 ms) : 0, 7425
Telemetry [candidate] (7.304 ms) : 0, 7304
IAST [baseline] (21.916 ms) : 0, 21916
IAST [candidate] (24.443 ms) : 0, 24443
section profiling
BytebuddyAgent [baseline] (666.036 ms) : 0, 666036
BytebuddyAgent [candidate] (666.071 ms) : 0, 666071
GlobalTracer [baseline] (391.52 ms) : 0, 391520
GlobalTracer [candidate] (389.704 ms) : 0, 389704
AppSec [baseline] (52.541 ms) : 0, 52541
AppSec [candidate] (52.431 ms) : 0, 52431
Remote Config [baseline] (707.877 µs) : 0, 708
Remote Config [candidate] (681.191 µs) : 0, 681
Telemetry [baseline] (7.473 ms) : 0, 7473
Telemetry [candidate] (7.364 ms) : 0, 7364
ProfilingAgent [baseline] (96.367 ms) : 0, 96367
ProfilingAgent [candidate] (96.522 ms) : 0, 96522
Profiling [baseline] (96.391 ms) : 0, 96391
Profiling [candidate] (96.546 ms) : 0, 96546
Loading
Startup time reports for insecure-bank
gantt
    title insecure-bank - global startup overhead: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.057 s) : 0, 1057428
Total [baseline] (8.505 s) : 0, 8504932
Agent [candidate] (1.052 s) : 0, 1052101
Total [candidate] (8.502 s) : 0, 8501804
section iast
Agent [baseline] (1.175 s) : 0, 1174804
Total [baseline] (8.961 s) : 0, 8961435
Agent [candidate] (1.178 s) : 0, 1177604
Total [candidate] (8.97 s) : 0, 8969965
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.174 s) : 0, 1173569
Total [baseline] (8.945 s) : 0, 8944599
Agent [candidate] (1.174 s) : 0, 1174165
Total [candidate] (8.953 s) : 0, 8952973
section iast_TELEMETRY_OFF
Agent [baseline] (1.173 s) : 0, 1173083
Total [baseline] (8.953 s) : 0, 8952532
Agent [candidate] (1.173 s) : 0, 1172822
Total [candidate] (9.004 s) : 0, 9003701
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.057 s -
Agent iast 1.175 s 117.376 ms (11.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.174 s 116.141 ms (11.0%)
Agent iast_TELEMETRY_OFF 1.173 s 115.654 ms (10.9%)
Total tracing 8.505 s -
Total iast 8.961 s 456.503 ms (5.4%)
Total iast_HARDCODED_SECRET_DISABLED 8.945 s 439.667 ms (5.2%)
Total iast_TELEMETRY_OFF 8.953 s 447.601 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.052 s -
Agent iast 1.178 s 125.503 ms (11.9%)
Agent iast_HARDCODED_SECRET_DISABLED 1.174 s 122.064 ms (11.6%)
Agent iast_TELEMETRY_OFF 1.173 s 120.72 ms (11.5%)
Total tracing 8.502 s -
Total iast 8.97 s 468.16 ms (5.5%)
Total iast_HARDCODED_SECRET_DISABLED 8.953 s 451.169 ms (5.3%)
Total iast_TELEMETRY_OFF 9.004 s 501.896 ms (5.9%)
gantt
    title insecure-bank - break down per module: candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.995 ms) : 0, 674995
BytebuddyAgent [candidate] (670.941 ms) : 0, 670941
GlobalTracer [baseline] (309.231 ms) : 0, 309231
GlobalTracer [candidate] (308.235 ms) : 0, 308235
AppSec [baseline] (51.401 ms) : 0, 51401
AppSec [candidate] (51.277 ms) : 0, 51277
Remote Config [baseline] (680.799 µs) : 0, 681
Remote Config [candidate] (690.27 µs) : 0, 690
Telemetry [baseline] (7.446 ms) : 0, 7446
Telemetry [candidate] (7.375 ms) : 0, 7375
section iast
BytebuddyAgent [baseline] (781.848 ms) : 0, 781848
BytebuddyAgent [candidate] (782.924 ms) : 0, 782924
GlobalTracer [baseline] (296.27 ms) : 0, 296270
GlobalTracer [candidate] (296.972 ms) : 0, 296972
AppSec [baseline] (54.207 ms) : 0, 54207
AppSec [candidate] (52.618 ms) : 0, 52618
Remote Config [baseline] (600.702 µs) : 0, 601
Remote Config [candidate] (600.476 µs) : 0, 600
Telemetry [baseline] (7.395 ms) : 0, 7395
Telemetry [candidate] (7.285 ms) : 0, 7285
IAST [baseline] (20.86 ms) : 0, 20860
IAST [candidate] (23.543 ms) : 0, 23543
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (779.235 ms) : 0, 779235
BytebuddyAgent [candidate] (780.088 ms) : 0, 780088
GlobalTracer [baseline] (296.361 ms) : 0, 296361
GlobalTracer [candidate] (296.021 ms) : 0, 296021
AppSec [baseline] (50.613 ms) : 0, 50613
AppSec [candidate] (53.009 ms) : 0, 53009
Remote Config [baseline] (597.541 µs) : 0, 598
Remote Config [candidate] (593.603 µs) : 0, 594
Telemetry [baseline] (7.264 ms) : 0, 7264
Telemetry [candidate] (8.108 ms) : 0, 8108
IAST [baseline] (25.915 ms) : 0, 25915
IAST [candidate] (22.74 ms) : 0, 22740
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (778.981 ms) : 0, 778981
BytebuddyAgent [candidate] (778.98 ms) : 0, 778980
GlobalTracer [baseline] (296.272 ms) : 0, 296272
GlobalTracer [candidate] (296.397 ms) : 0, 296397
AppSec [baseline] (54.194 ms) : 0, 54194
AppSec [candidate] (53.756 ms) : 0, 53756
Remote Config [baseline] (591.14 µs) : 0, 591
Remote Config [candidate] (599.96 µs) : 0, 600
Telemetry [baseline] (7.244 ms) : 0, 7244
Telemetry [candidate] (8.017 ms) : 0, 8017
IAST [baseline] (22.178 ms) : 0, 22178
IAST [candidate] (21.44 ms) : 0, 21440
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-09-09T16:09:18 2024-09-09T16:16:04
git_branch master nikita-tkachenko/code-coverage-with-itr
git_commit_date 1725890539 1725897527
git_commit_sha 1d59e61 c0c0f53
release_version 1.40.0-SNAPSHOT~1d59e612a6 1.40.0-SNAPSHOT~c0c0f5394f
start_time 2024-09-09T16:09:05 2024-09-09T16:15:51
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1725898908 1725898908
ci_job_id 632289989 632289989
ci_pipeline_id 43877319 43877319
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 19 unstable metrics.

Request duration reports for petclinic
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.332 ms) : 1314, 1351
.   : milestone, 1332,
appsec (1.738 ms) : 1715, 1762
.   : milestone, 1738,
appsec_no_iast (1.741 ms) : 1717, 1766
.   : milestone, 1741,
iast (1.472 ms) : 1450, 1494
.   : milestone, 1472,
profiling (1.468 ms) : 1444, 1491
.   : milestone, 1468,
tracing (1.479 ms) : 1455, 1502
.   : milestone, 1479,
section candidate
no_agent (1.348 ms) : 1329, 1366
.   : milestone, 1348,
appsec (1.732 ms) : 1708, 1756
.   : milestone, 1732,
appsec_no_iast (1.73 ms) : 1705, 1755
.   : milestone, 1730,
iast (1.49 ms) : 1466, 1513
.   : milestone, 1490,
profiling (1.468 ms) : 1445, 1491
.   : milestone, 1468,
tracing (1.466 ms) : 1442, 1491
.   : milestone, 1466,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.332 ms [1.314 ms, 1.351 ms] -
appsec 1.738 ms [1.715 ms, 1.762 ms] 405.788 µs (30.5%)
appsec_no_iast 1.741 ms [1.717 ms, 1.766 ms] 408.989 µs (30.7%)
iast 1.472 ms [1.45 ms, 1.494 ms] 139.566 µs (10.5%)
profiling 1.468 ms [1.444 ms, 1.491 ms] 135.217 µs (10.1%)
tracing 1.479 ms [1.455 ms, 1.502 ms] 146.169 µs (11.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.348 ms [1.329 ms, 1.366 ms] -
appsec 1.732 ms [1.708 ms, 1.756 ms] 384.034 µs (28.5%)
appsec_no_iast 1.73 ms [1.705 ms, 1.755 ms] 382.831 µs (28.4%)
iast 1.49 ms [1.466 ms, 1.513 ms] 142.326 µs (10.6%)
profiling 1.468 ms [1.445 ms, 1.491 ms] 120.407 µs (8.9%)
tracing 1.466 ms [1.442 ms, 1.491 ms] 118.939 µs (8.8%)
Request duration reports for insecure-bank
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
    dateFormat X
    axisFormat %s
section baseline
no_agent (368.205 µs) : 349, 388
.   : milestone, 368,
iast (474.487 µs) : 453, 496
.   : milestone, 474,
iast_FULL (541.34 µs) : 520, 563
.   : milestone, 541,
iast_GLOBAL (495.847 µs) : 474, 517
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (476.845 µs) : 454, 500
.   : milestone, 477,
iast_INACTIVE (436.534 µs) : 416, 457
.   : milestone, 437,
iast_TELEMETRY_OFF (473.319 µs) : 451, 496
.   : milestone, 473,
tracing (432.274 µs) : 412, 452
.   : milestone, 432,
section candidate
no_agent (363.587 µs) : 344, 383
.   : milestone, 364,
iast (477.028 µs) : 455, 499
.   : milestone, 477,
iast_FULL (548.945 µs) : 528, 570
.   : milestone, 549,
iast_GLOBAL (506.534 µs) : 485, 528
.   : milestone, 507,
iast_HARDCODED_SECRET_DISABLED (478.238 µs) : 456, 500
.   : milestone, 478,
iast_INACTIVE (439.448 µs) : 418, 461
.   : milestone, 439,
iast_TELEMETRY_OFF (466.839 µs) : 444, 490
.   : milestone, 467,
tracing (431.968 µs) : 412, 452
.   : milestone, 432,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.205 µs [348.554 µs, 387.856 µs] -
iast 474.487 µs [452.856 µs, 496.118 µs] 106.282 µs (28.9%)
iast_FULL 541.34 µs [520.077 µs, 562.604 µs] 173.135 µs (47.0%)
iast_GLOBAL 495.847 µs [474.481 µs, 517.213 µs] 127.642 µs (34.7%)
iast_HARDCODED_SECRET_DISABLED 476.845 µs [454.094 µs, 499.596 µs] 108.64 µs (29.5%)
iast_INACTIVE 436.534 µs [416.117 µs, 456.95 µs] 68.329 µs (18.6%)
iast_TELEMETRY_OFF 473.319 µs [450.599 µs, 496.04 µs] 105.114 µs (28.5%)
tracing 432.274 µs [412.253 µs, 452.296 µs] 64.069 µs (17.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.587 µs [344.029 µs, 383.146 µs] -
iast 477.028 µs [455.032 µs, 499.024 µs] 113.441 µs (31.2%)
iast_FULL 548.945 µs [527.606 µs, 570.283 µs] 185.358 µs (51.0%)
iast_GLOBAL 506.534 µs [484.757 µs, 528.311 µs] 142.947 µs (39.3%)
iast_HARDCODED_SECRET_DISABLED 478.238 µs [456.312 µs, 500.163 µs] 114.651 µs (31.5%)
iast_INACTIVE 439.448 µs [418.269 µs, 460.626 µs] 75.86 µs (20.9%)
iast_TELEMETRY_OFF 466.839 µs [444.15 µs, 489.528 µs] 103.252 µs (28.4%)
tracing 431.968 µs [411.541 µs, 452.394 µs] 68.381 µs (18.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master nikita-tkachenko/code-coverage-with-itr
git_commit_date 1725890539 1725897527
git_commit_sha 1d59e61 c0c0f53
release_version 1.40.0-SNAPSHOT~1d59e612a6 1.40.0-SNAPSHOT~c0c0f5394f
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1725899477 1725899477
ci_job_id 632289990 632289990
ci_pipeline_id 43877319 43877319
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.453 ms) : 1442, 1464
.   : milestone, 1453,
appsec (2.217 ms) : 2182, 2252
.   : milestone, 2217,
iast (1.978 ms) : 1935, 2021
.   : milestone, 1978,
iast_GLOBAL (2.036 ms) : 1991, 2081
.   : milestone, 2036,
profiling (1.857 ms) : 1823, 1892
.   : milestone, 1857,
tracing (1.852 ms) : 1819, 1886
.   : milestone, 1852,
section candidate
no_agent (1.455 ms) : 1444, 1466
.   : milestone, 1455,
appsec (2.248 ms) : 2212, 2283
.   : milestone, 2248,
iast (1.98 ms) : 1937, 2023
.   : milestone, 1980,
iast_GLOBAL (2.034 ms) : 1990, 2079
.   : milestone, 2034,
profiling (1.863 ms) : 1829, 1897
.   : milestone, 1863,
tracing (1.836 ms) : 1803, 1869
.   : milestone, 1836,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.453 ms [1.442 ms, 1.464 ms] -
appsec 2.217 ms [2.182 ms, 2.252 ms] 764.598 µs (52.6%)
iast 1.978 ms [1.935 ms, 2.021 ms] 525.531 µs (36.2%)
iast_GLOBAL 2.036 ms [1.991 ms, 2.081 ms] 583.239 µs (40.1%)
profiling 1.857 ms [1.823 ms, 1.892 ms] 404.617 µs (27.8%)
tracing 1.852 ms [1.819 ms, 1.886 ms] 399.51 µs (27.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.455 ms [1.444 ms, 1.466 ms] -
appsec 2.248 ms [2.212 ms, 2.283 ms] 792.396 µs (54.5%)
iast 1.98 ms [1.937 ms, 2.023 ms] 524.603 µs (36.1%)
iast_GLOBAL 2.034 ms [1.99 ms, 2.079 ms] 579.072 µs (39.8%)
profiling 1.863 ms [1.829 ms, 1.897 ms] 407.91 µs (28.0%)
tracing 1.836 ms [1.803 ms, 1.869 ms] 380.537 µs (26.2%)
Execution time for biojava
gantt
    title biojava - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~c0c0f5394f, baseline=1.40.0-SNAPSHOT~1d59e612a6
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.602 s) : 15602000, 15602000
.   : milestone, 15602000,
appsec (15.182 s) : 15182000, 15182000
.   : milestone, 15182000,
iast (18.811 s) : 18811000, 18811000
.   : milestone, 18811000,
iast_GLOBAL (17.809 s) : 17809000, 17809000
.   : milestone, 17809000,
profiling (15.924 s) : 15924000, 15924000
.   : milestone, 15924000,
tracing (14.855 s) : 14855000, 14855000
.   : milestone, 14855000,
section candidate
no_agent (15.38 s) : 15380000, 15380000
.   : milestone, 15380000,
appsec (15.16 s) : 15160000, 15160000
.   : milestone, 15160000,
iast (18.957 s) : 18957000, 18957000
.   : milestone, 18957000,
iast_GLOBAL (18.121 s) : 18121000, 18121000
.   : milestone, 18121000,
profiling (15.776 s) : 15776000, 15776000
.   : milestone, 15776000,
tracing (14.829 s) : 14829000, 14829000
.   : milestone, 14829000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.602 s [15.602 s, 15.602 s] -
appsec 15.182 s [15.182 s, 15.182 s] -420.0 ms (-2.7%)
iast 18.811 s [18.811 s, 18.811 s] 3.209 s (20.6%)
iast_GLOBAL 17.809 s [17.809 s, 17.809 s] 2.207 s (14.1%)
profiling 15.924 s [15.924 s, 15.924 s] 322.0 ms (2.1%)
tracing 14.855 s [14.855 s, 14.855 s] -747.0 ms (-4.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.38 s [15.38 s, 15.38 s] -
appsec 15.16 s [15.16 s, 15.16 s] -220.0 ms (-1.4%)
iast 18.957 s [18.957 s, 18.957 s] 3.577 s (23.3%)
iast_GLOBAL 18.121 s [18.121 s, 18.121 s] 2.741 s (17.8%)
profiling 15.776 s [15.776 s, 15.776 s] 396.0 ms (2.6%)
tracing 14.829 s [14.829 s, 14.829 s] -551.0 ms (-3.6%)

@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch from 7852e93 to 4a40e16 Compare August 21, 2024 13:58
@github-actions

Copy link
Copy Markdown
Contributor

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

Hi! 👋 Looks like you updated a Git Submodule.
If this was not intentional please make sure to:

@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch 4 times, most recently from fdfc064 to 4530a48 Compare August 28, 2024 13:12

@Nonnull
public byte[] getCoverageData() {
return coverageData;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Code Quality Violation

Do not return an array that is a class member (...read more)

Avoid returning an array that was defined as a class member. If you want to return an array or the values of the array that is a class member, consider returning a copy of the array.

View in Datadog  Leave us feedback  Documentation


@Override
public boolean equals(Object o) {
if (this == o) return true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Quality Violation

Suggested change
if (this == o) return true;
if (this == o) {return true};
single if statement should be wrapped in a brace (...read more)

Omitting braces {} is valid in multiple statements, such as, for loops, if statements, and while loops. However, enforcing the use of control braces throughout your codebase will make the code more consistent and can make it easier to add statements in the future.

View in Datadog  Leave us feedback  Documentation

@Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Quality Violation

Suggested change
if (o == null || getClass() != o.getClass()) return false;
if (o == null || getClass() != o.getClass()) {return false};
single if statement should be wrapped in a brace (...read more)

Omitting braces {} is valid in multiple statements, such as, for loops, if statements, and while loops. However, enforcing the use of control braces throughout your codebase will make the code more consistent and can make it easier to add statements in the future.

View in Datadog  Leave us feedback  Documentation

@nikita-tkachenko-datadog nikita-tkachenko-datadog changed the title Refactoring to prepare for ITR code coverage ITR code coverage support Sep 6, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog marked this pull request as ready for review September 6, 2024 14:52
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch from db565fd to 7fa7ab4 Compare September 6, 2024 17:10
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch from 7fa7ab4 to 56bc06d Compare September 9, 2024 09:44
@nikita-tkachenko-datadog
nikita-tkachenko-datadog force-pushed the nikita-tkachenko/code-coverage-with-itr branch from 56bc06d to c0c0f53 Compare September 9, 2024 15:58
@nikita-tkachenko-datadog
nikita-tkachenko-datadog merged commit ff190af into master Sep 10, 2024
@nikita-tkachenko-datadog
nikita-tkachenko-datadog deleted the nikita-tkachenko/code-coverage-with-itr branch September 10, 2024 10:20
@github-actions github-actions Bot added this to the 1.40.0 milestone Sep 10, 2024
tonyredondo added a commit to DataDog/dd-trace-dotnet that referenced this pull request Jun 17, 2026
## Summary of changes

- Implements ITR code coverage backfill for .NET test runs that skip
tests while a line-coverage source is active.
- Parses backend `meta.coverage` bitmaps from skippable-tests responses
using the .NET `FileBitmap` representation and persists scoped backfill
data across child processes, injected sessions, AppDomains, and coverage
tool adapters.
- Backfills line coverage for Datadog internal coverage, Coverlet
in-memory results, Coverlet collector XML attachments,
Cobertura/OpenCover XML, and line-capable Microsoft CodeCoverage XML
reports.
- Rewrites supported customer-visible report files before downstream
coverage tools read their final results.
- Publishes one session-level `test.code_coverage.lines_pct` result
through source arbitration, with `test.code_coverage.backfilled` when
the selected result used backend-aware coverage data.
- Keeps ITR skip behavior aligned with the Java reference
implementation: when line coverage is active, a skippable test is denied
only when the backend marks that candidate with
`_missing_line_code_coverage=true`. Missing or invalid aggregate
`meta.coverage` disables safe backfill publication, but does not
globally disable ITR skipping.
- Sends custom test configuration tags to the backend and trusts the
backend response, matching Java. The tracer does not locally filter
skippable tests by `custom` configurations.

## Reason for change

ITR can skip tests that would otherwise contribute line coverage.
Without backfill, coverage tools and the Datadog session metric can
under-report line coverage for a run where skipped tests were
intentionally omitted. The backend now returns skipped-test line
coverage in the skippable-tests response, so the tracer can merge
backend line bitmaps with local coverage before publishing the session
result or rewriting external reports.

References:

- RFC:
https://docs.google.com/document/d/1234bLTvL-v9PzA3zIAf_2NN8DdTpfLM8NX4WiniPssw/edit?tab=t.0#heading=h.rnd972k0hiye
- Java reference implementation:
DataDog/dd-trace-java#7367

## Java compatibility notes

This PR follows the Java reference behavior for the user-visible ITR
decision:

- `custom` test configurations are sent to the backend, but are not used
for local post-filtering.
- `test.bundle`/module scope is used to associate backend candidates
with the current module scope.
- `_missing_line_code_coverage=true` prevents skipping that candidate
when line coverage is active.
- Missing, absent, or invalid aggregate `meta.coverage` does not prevent
ITR from skipping tests. It only means the .NET tracer must avoid
publishing a backfilled coverage result that cannot be proven safe.

The intentional implementation difference is bitmap representation: Java
decodes backend coverage into `BitSet`, while .NET keeps the existing
`FileBitmap` representation for both decoding backend coverage and
sending/persisting coverage data.

## Review diagrams

### End-to-end backfill flow

```mermaid
flowchart TD
  A["dotnet test / vstest / coverage command starts"]
  B["CoverageBackfillCapability inspects command, runsettings, response files, and report outputs"]
  C{"Line-capable coverage source active?"}
  D["Legacy ITR behavior; no backfill required"]
  E["Request scoped skippable tests for the current test.bundle/module"]
  F["Backend returns skippable candidates, _missing_line_code_coverage flags, and optional meta.coverage"]
  G["Java-compatible skip gate decides whether each candidate may be skipped"]
  H["Framework integration records the actual ITR skip only after it commits to skipping"]
  I["CoverageBackfillData decodes meta.coverage with FileBitmap"]
  J["CoverageBackfillDataStore persists backend coverage, actual-skip state, scoped backfillable state, IPC results, and IPC failures"]
  K["Coverage producers apply or rewrite line coverage results"]
  L["Validation checks paths, represented backend lines, producer completeness, and IPC state"]
  M["CodeCoverageResultAggregator selects one publishable source by priority"]
  N["Session publishes test.code_coverage.lines_pct and test.code_coverage.backfilled when applicable"]

  A --> B --> C
  C -- "no" --> D
  C -- "yes" --> E --> F
  F --> G --> H --> J
  F --> I --> J
  J --> K --> L --> M --> N
```

### Skippable scope and Java-compatible skip gate

```mermaid
flowchart TD
  A["Skippable-tests response item"]
  B{"Top-level test.bundle mismatches scoped request?"}
  C["Drop only this mismatched candidate"]
  D["Keep candidate; do not filter by custom configs"]
  E{"Line coverage active?"}
  F["Use legacy ITR skip behavior"]
  G{"Candidate has _missing_line_code_coverage=true?"}
  H["Do not skip this candidate"]
  I["Allow ITR skip"]
  J["meta.coverage is decoded separately and is not part of the skip decision"]

  A --> B
  B -- "yes" --> C
  B -- "no" --> D
  D --> E
  E -- "no" --> F --> I
  E -- "yes" --> G
  G -- "yes" --> H
  G -- "no or absent" --> I
  J -. "matches Java" .-> G
```

### Coverage producers, persistence, and publication

```mermaid
flowchart LR
  A["Backend meta.coverage FileBitmap data"]
  B["Actual ITR skip markers"]
  C["CoverageBackfillDataStore"]

  subgraph Producers["Coverage producers"]
    D["Datadog internal coverage"]
    E["Coverlet in-memory result"]
    F["Coverlet collector XML fallback"]
    G["Cobertura/OpenCover XML"]
    H["Microsoft CodeCoverage XML"]
  end

  I["Apply backend line bitmaps or rewrite customer-visible XML"]
  J["Send live IPC result or persist result/failure marker"]
  K["Validate source paths, represented lines, partial merges, and IPC completeness"]
  L["Suppress stale or unsafe results"]
  M["Select highest-priority publishable source"]
  N["Publish final session coverage tags"]

  A --> C
  B --> C
  C --> Producers
  Producers --> I --> J --> K
  K -- "safe" --> M --> N
  K -- "unsafe" --> L
```

## Implementation details

- Adds `CoverageBackfillData` parsing for backend `meta.coverage`,
including bitmap decoding, validation, merging, and source-path
normalization.
- Adds `CoverageBackfillDataStore` to persist backend coverage,
actual-skip markers, backfillable scoped skip state, coverage IPC
results, and coverage IPC failures across process and AppDomain
boundaries.
- Adds scoped skippable-tests requests when coverage backfill is
required, using `test.bundle` plus runtime/configuration fingerprinting
so backend coverage matches the current local execution scope.
- Separates skippable candidates from tests actually skipped by ITR.
Framework integrations record coverage-backfill skip state when they
commit to an ITR skip; `Test.Close` records the session-level "ITR skip
happened" tag state later.
- Uses Java-compatible skip gating: the skip decision checks the backend
candidate's `_missing_line_code_coverage` flag, while aggregate backend
coverage validity is tracked separately for deciding whether a
backfilled coverage result can be published.
- Adds `CoverageBackfillCapability` and command-line parsing for
activation checks across `dotnet test`, `vstest`, `dotnet-coverage`,
Coverlet collector/MSBuild, Microsoft CodeCoverage, response files,
runsettings, MSBuild properties, filters, framework selectors,
thresholds, and generated report paths.
- Adds shared path matching and validation state so partial coverage
reports can only publish when their merged result represents all
required backend skipped-test lines and did not rely on unsafe local
suffix matches.
- Adds adapters for Datadog internal global coverage, Coverlet modules,
Coverlet collector XML fallback, external Cobertura/OpenCover XML, and
Microsoft CodeCoverage XML.
- Adds `CodeCoverageResultAggregator` to select the final session result
by source priority and suppress ambiguous count-based same-source
merges.

## Key behavior cases

- If no coverage report source is active, ITR keeps the legacy skip
behavior and no coverage backfill is applied.
- If line coverage is active, a skippable test is not skipped only when
the backend marks that candidate as missing line coverage.
- If backend aggregate `meta.coverage` is missing or invalid, ITR can
still skip tests, but the session will not publish a backfilled coverage
result unless another validated producer result is available.
- If coverage is active but no test is actually skipped by ITR, reports
can still be read normally and backend backfill is not applied.
- If an external XML report is selected, it has higher priority than
internal or IPC-based coverage results.
- If a child coverage tool cannot deliver a complete result when
backfill publication is required, the session fails closed for coverage
publication instead of publishing stale lower-priority coverage.
- Branch, method, condition, complexity, and threshold metadata are
preserved but not reconstructed from backend data; backend coverage is
line bitmap data only.

## Source priority

`CodeCoverageResultAggregator` selects the highest-priority publishable
result:

1. `ExternalXml`
2. `CoverletXmlFallback`
3. `Coverlet`
4. `MicrosoftCodeCoverage`
5. `Unknown`
6. `DatadogInternal`

Same-source results with line counts are only publishable when they were
explicitly merged by the producer or can be validated without ambiguous
count aggregation.

## Type guide

| Type | Responsibility |
| --- | --- |
| `CoverageBackfillData` | Decodes and validates backend `meta.coverage`
bitmaps. |
| `CoverageBackfillDataStore` | Persists backend coverage, actual-skip
state, scoped backfillable skip state, IPC results, and IPC failures. |
| `CoverageBackfillCapability` | Determines whether backfill is required
and whether the active coverage command/report path is safe. |
| `CoverageBackfillCommandLine` | Parses command lines, response files,
runsettings, MSBuild properties, and coverage output options for
capability checks. |
| `CoverageBackfillPathMatcher` / `CoverageBackfillPathMatchTracker` |
Matches backend source paths to local report paths and detects unsafe
suffix ambiguity. |
| `CoverageBackfillApplicator` | Applies backend line bitmaps to Datadog
internal global coverage. |
| `ExternalCoverageXmlBackfill` | Parses, validates, rewrites, and
merges line-capable XML reports. |
| `CoverletCoverageBackfill` | Mutates Coverlet's in-memory modules
before Coverlet calculates its line summary. |
| `CodeCoverageBackfillValidation` | Carries cross-result validation
state so partial producer results only publish when the merged set is
complete and safe. |
| `TestOptimizationSkippableFeature` | Requests scoped skippable tests,
gates coverage-active skips, and records actual ITR skip state. |
| `SessionCodeCoverageMessage` | Sends source-specific coverage results
from child tools/domains to the parent test session. |
| `CodeCoverageResultAggregator` | Suppresses unsafe results, merges
validated producer output, and selects the final session coverage source
by priority. |

## Test coverage

Added and expanded coverage across:

- backend `meta.coverage` parsing, bitmap merging, invalid payload
handling, missing coverage, and missing-line metadata;
- Java-compatible skippable parsing behavior for `custom` configurations
and `test.bundle` module scope;
- scoped skippable requests, actual-skip tracking, bundle/config
fingerprinting, and coverage-publication safety state;
- Datadog internal coverage backfill and `LineCallCount` semantics;
- Coverlet module mutation, rollback, validation, partial result
merging, and IPC failure paths;
- Cobertura, OpenCover, Microsoft CodeCoverage XML parsing/rewrite/merge
paths;
- path normalization, suffix ambiguity detection, source-root ambiguity,
represented-line validation, and multi-report validation;
- `dotnet test`, VSTest, Coverlet, Microsoft CodeCoverage,
`dotnet-coverage`, response file, runsettings, and MSBuild command
detection;
- session result arbitration, persisted IPC recovery, atomic
marker/result races, late Coverlet XML fallback, and injected session
behavior.

Recent validation:

- Java-parity focused unit tests for skippable parsing, skip gating, and
backend coverage decoding passed on `net10.0`.
- `AttributeTests.AttributesInstantiationsOnlyUseBuiltinTypes` passed on
`net10.0` and `net9.0` x64.
- `IpcTests.IpcClientCanSendCoverageMessageWithBackfillValidation`
passed on `net10.0`.
-
`ManagedVanguardStopIntegrationTests.DeduplicatesEquivalentMicrosoftLineXmlPathsBeforeBackfill`
passed on `net10.0`.
- `git diff --check`: clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: ci visibility Continuous Integration Visibility run-tests: all Run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants