Skip to content

Tests summary in pipeline and improved test tags to be used in Test Optimization product#10530

Merged
bric3 merged 21 commits intomasterfrom
bdu/total-tests
Feb 11, 2026
Merged

Tests summary in pipeline and improved test tags to be used in Test Optimization product#10530
bric3 merged 21 commits intomasterfrom
bdu/total-tests

Conversation

@bric3
Copy link
Copy Markdown
Contributor

@bric3 bric3 commented Feb 5, 2026 •
edited
Loading

What Does This Do

Compute ran tests in a pipeline, e.g.

Overall Summary

Metric Count
Total Tests 284627
Passed 237614
Failed 0
Skipped 47013

Breakdown by JVM Version

JVM Version Total Tests Passed Failed Skipped Job Count
8 56996 47575 0 9421 27
semeru8 1064 993 0 71 2
11 42908 37096 0 5812 20
17 61314 50719 0 10595 26
graalvm17 3 1 0 2 1
21 61215 50680 0 10535 26
graalvm21 3 1 0 2 1
25 61121 50548 0 10573 26
graalvm25 3 1 0 2 1

Not some sections can be expanded

image

Also, this PR push new tags to JUnit report allow easier querying in the TO interface. This query for example, to get the same picture on tests

image

Motivation

Having a holistic picture of how many tests passed.

If the numbers remain stable for any PR, this could serve as a basis for a possible merge gate.

Additional Notes

@bric3 bric3 requested a review from a team as a code owner February 5, 2026 14:17
@bric3 bric3 requested review from randomanderson and removed request for a team February 5, 2026 14:17
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Feb 5, 2026 •
edited
Loading

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@bric3 bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Feb 5, 2026
@bric3 bric3 changed the title Compute ran tests in a PR Compute ran tests in a pipeline Feb 5, 2026
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Feb 5, 2026 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master bdu/total-tests
git_commit_date 1770829518 1770842148
git_commit_sha 63b528a fecd36e
release_version 1.60.0-SNAPSHOT~63b528a901 1.60.0-SNAPSHOT~fecd36eaa8
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1770843995 1770843995
ci_job_id 1419103736 1419103736
ci_pipeline_id 95942597 95942597
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-phlbuk0s 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-phlbuk0s 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 57 metrics, 14 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.063 s) : 0, 1063367
Total [baseline] (8.746 s) : 0, 8745876
Agent [candidate] (1.073 s) : 0, 1073017
Total [candidate] (8.794 s) : 0, 8794182
section iast
Agent [baseline] (1.234 s) : 0, 1233919
Total [baseline] (9.379 s) : 0, 9379325
Agent [candidate] (1.25 s) : 0, 1249791
Total [candidate] (9.404 s) : 0, 9403666
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.063 s -
Agent iast 1.234 s 170.552 ms (16.0%)
Total tracing 8.746 s -
Total iast 9.379 s 633.449 ms (7.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.073 s -
Agent iast 1.25 s 176.774 ms (16.5%)
Total tracing 8.794 s -
Total iast 9.404 s 609.483 ms (6.9%) 
gantt
    title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.177 ms) : 0, 1177
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (628.564 ms) : 0, 628564
BytebuddyAgent [candidate] (634.017 ms) : 0, 634017
AgentMeter [baseline] (29.03 ms) : 0, 29030
AgentMeter [candidate] (29.184 ms) : 0, 29184
GlobalTracer [baseline] (257.286 ms) : 0, 257286
GlobalTracer [candidate] (259.756 ms) : 0, 259756
AppSec [baseline] (32.715 ms) : 0, 32715
AppSec [candidate] (33.096 ms) : 0, 33096
Debugger [baseline] (61.236 ms) : 0, 61236
Debugger [candidate] (62.492 ms) : 0, 62492
Remote Config [baseline] (624.81 µs) : 0, 625
Remote Config [candidate] (616.527 µs) : 0, 617
Telemetry [baseline] (12.178 ms) : 0, 12178
Telemetry [candidate] (11.698 ms) : 0, 11698
Flare Poller [baseline] (5.27 ms) : 0, 5270
Flare Poller [candidate] (5.409 ms) : 0, 5409
section iast
crashtracking [baseline] (1.21 ms) : 0, 1210
crashtracking [candidate] (1.198 ms) : 0, 1198
BytebuddyAgent [baseline] (797.943 ms) : 0, 797943
BytebuddyAgent [candidate] (809.285 ms) : 0, 809285
AgentMeter [baseline] (11.254 ms) : 0, 11254
AgentMeter [candidate] (11.431 ms) : 0, 11431
GlobalTracer [baseline] (249.091 ms) : 0, 249091
GlobalTracer [candidate] (251.375 ms) : 0, 251375
IAST [baseline] (27.134 ms) : 0, 27134
IAST [candidate] (27.365 ms) : 0, 27365
AppSec [baseline] (33.065 ms) : 0, 33065
AppSec [candidate] (36.081 ms) : 0, 36081
Debugger [baseline] (66.235 ms) : 0, 66235
Debugger [candidate] (64.534 ms) : 0, 64534
Remote Config [baseline] (531.659 µs) : 0, 532
Remote Config [candidate] (542.641 µs) : 0, 543
Telemetry [baseline] (8.659 ms) : 0, 8659
Telemetry [candidate] (8.659 ms) : 0, 8659
Flare Poller [baseline] (3.48 ms) : 0, 3480
Flare Poller [candidate] (3.517 ms) : 0, 3517
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.072 s) : 0, 1071892
Total [baseline] (10.866 s) : 0, 10866484
Agent [candidate] (1.074 s) : 0, 1073710
Total [candidate] (10.942 s) : 0, 10941750
section appsec
Agent [baseline] (1.244 s) : 0, 1243763
Total [baseline] (11.139 s) : 0, 11139487
Agent [candidate] (1.254 s) : 0, 1253595
Total [candidate] (11.148 s) : 0, 11147822
section iast
Agent [baseline] (1.241 s) : 0, 1240604
Total [baseline] (11.323 s) : 0, 11323271
Agent [candidate] (1.241 s) : 0, 1240830
Total [candidate] (11.204 s) : 0, 11203819
section profiling
Agent [baseline] (1.199 s) : 0, 1199285
Total [baseline] (10.444 s) : 0, 10444222
Agent [candidate] (1.204 s) : 0, 1204087
Total [candidate] (11.071 s) : 0, 11071418
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.072 s -
Agent appsec 1.244 s 171.871 ms (16.0%)
Agent iast 1.241 s 168.712 ms (15.7%)
Agent profiling 1.199 s 127.393 ms (11.9%)
Total tracing 10.866 s -
Total appsec 11.139 s 273.003 ms (2.5%)
Total iast 11.323 s 456.787 ms (4.2%)
Total profiling 10.444 s -422.262 ms (-3.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.074 s -
Agent appsec 1.254 s 179.885 ms (16.8%)
Agent iast 1.241 s 167.12 ms (15.6%)
Agent profiling 1.204 s 130.377 ms (12.1%)
Total tracing 10.942 s -
Total appsec 11.148 s 206.073 ms (1.9%)
Total iast 11.204 s 262.069 ms (2.4%)
Total profiling 11.071 s 129.668 ms (1.2%) 
gantt
    title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.188 ms) : 0, 1188
crashtracking [candidate] (1.193 ms) : 0, 1193
BytebuddyAgent [baseline] (633.819 ms) : 0, 633819
BytebuddyAgent [candidate] (634.195 ms) : 0, 634195
AgentMeter [baseline] (29.183 ms) : 0, 29183
AgentMeter [candidate] (29.217 ms) : 0, 29217
GlobalTracer [baseline] (259.159 ms) : 0, 259159
GlobalTracer [candidate] (259.673 ms) : 0, 259673
AppSec [baseline] (32.974 ms) : 0, 32974
AppSec [candidate] (32.969 ms) : 0, 32969
Debugger [baseline] (63.181 ms) : 0, 63181
Debugger [candidate] (63.217 ms) : 0, 63217
Remote Config [baseline] (645.488 µs) : 0, 645
Remote Config [candidate] (630.581 µs) : 0, 631
Telemetry [baseline] (9.378 ms) : 0, 9378
Telemetry [candidate] (12.29 ms) : 0, 12290
Flare Poller [baseline] (6.912 ms) : 0, 6912
Flare Poller [candidate] (4.588 ms) : 0, 4588
section appsec
crashtracking [baseline] (1.186 ms) : 0, 1186
crashtracking [candidate] (1.202 ms) : 0, 1202
BytebuddyAgent [baseline] (660.613 ms) : 0, 660613
BytebuddyAgent [candidate] (668.544 ms) : 0, 668544
AgentMeter [baseline] (12.006 ms) : 0, 12006
AgentMeter [candidate] (12.101 ms) : 0, 12101
GlobalTracer [baseline] (259.026 ms) : 0, 259026
GlobalTracer [candidate] (261.027 ms) : 0, 261027
IAST [baseline] (25.278 ms) : 0, 25278
IAST [candidate] (25.374 ms) : 0, 25374
AppSec [baseline] (168.409 ms) : 0, 168409
AppSec [candidate] (168.442 ms) : 0, 168442
Debugger [baseline] (67.482 ms) : 0, 67482
Debugger [candidate] (67.213 ms) : 0, 67213
Remote Config [baseline] (659.041 µs) : 0, 659
Remote Config [candidate] (670.725 µs) : 0, 671
Telemetry [baseline] (9.708 ms) : 0, 9708
Telemetry [candidate] (9.479 ms) : 0, 9479
Flare Poller [baseline] (3.826 ms) : 0, 3826
Flare Poller [candidate] (3.902 ms) : 0, 3902
section iast
crashtracking [baseline] (1.199 ms) : 0, 1199
crashtracking [candidate] (1.196 ms) : 0, 1196
BytebuddyAgent [baseline] (800.673 ms) : 0, 800673
BytebuddyAgent [candidate] (800.479 ms) : 0, 800479
AgentMeter [baseline] (11.306 ms) : 0, 11306
AgentMeter [candidate] (11.341 ms) : 0, 11341
GlobalTracer [baseline] (250.403 ms) : 0, 250403
GlobalTracer [candidate] (250.474 ms) : 0, 250474
IAST [baseline] (27.048 ms) : 0, 27048
IAST [candidate] (27.21 ms) : 0, 27210
AppSec [baseline] (34.204 ms) : 0, 34204
AppSec [candidate] (31.501 ms) : 0, 31501
Debugger [baseline] (67.564 ms) : 0, 67564
Debugger [candidate] (70.387 ms) : 0, 70387
Remote Config [baseline] (551.75 µs) : 0, 552
Remote Config [candidate] (557.108 µs) : 0, 557
Telemetry [baseline] (8.742 ms) : 0, 8742
Telemetry [candidate] (8.652 ms) : 0, 8652
Flare Poller [baseline] (3.466 ms) : 0, 3466
Flare Poller [candidate] (3.525 ms) : 0, 3525
section profiling
crashtracking [baseline] (1.235 ms) : 0, 1235
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (687.068 ms) : 0, 687068
BytebuddyAgent [candidate] (690.726 ms) : 0, 690726
AgentMeter [baseline] (8.677 ms) : 0, 8677
AgentMeter [candidate] (8.629 ms) : 0, 8629
GlobalTracer [baseline] (217.582 ms) : 0, 217582
GlobalTracer [candidate] (218.269 ms) : 0, 218269
AppSec [baseline] (32.773 ms) : 0, 32773
AppSec [candidate] (32.893 ms) : 0, 32893
Debugger [baseline] (67.909 ms) : 0, 67909
Debugger [candidate] (68.127 ms) : 0, 68127
Remote Config [baseline] (624.427 µs) : 0, 624
Remote Config [candidate] (637.224 µs) : 0, 637
Telemetry [baseline] (8.962 ms) : 0, 8962
Telemetry [candidate] (9.036 ms) : 0, 9036
Flare Poller [baseline] (3.75 ms) : 0, 3750
Flare Poller [candidate] (3.744 ms) : 0, 3744
ProfilingAgent [baseline] (100.381 ms) : 0, 100381
ProfilingAgent [candidate] (100.243 ms) : 0, 100243
Profiling [baseline] (100.971 ms) : 0, 100971
Profiling [candidate] (100.817 ms) : 0, 100817
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master bdu/total-tests
git_commit_date 1770829518 1770842148
git_commit_sha 63b528a fecd36e
release_version 1.60.0-SNAPSHOT~63b528a901 1.60.0-SNAPSHOT~fecd36eaa8
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1770844495 1770844495
ci_job_id 1419103737 1419103737
ci_pipeline_id 95942597 95942597
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-ka1zg2pb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-ka1zg2pb 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 3 performance regressions! Performance is the same for 15 metrics, 17 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:profiling:high_load worse
[+61.577µs; +170.125µs] or [+3.846%; +10.626%]		 unstable
[+435.728µs; +1324.049µs] or [+9.657%; +29.343%]		 unstable
[-524.040op/s; -2.835op/s] or [-22.746%; -0.123%]		 1.717ms 5.392ms 2040.469op/s 1.601ms 4.512ms 2303.906op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-195.143µs; -107.154µs] or [-6.834%; -3.752%]		 same
[-346.133µs; +222.087µs] or [-4.358%; +2.796%]		 unstable
[-92.756op/s; +187.631op/s] or [-7.287%; +14.741%]		 2.704ms 7.881ms 1320.281op/s 2.856ms 7.943ms 1272.844op/s
scenario:load:petclinic:iast:high_load worse
[+0.773ms; +1.526ms] or [+4.567%; +9.011%]		 worse
[+0.683ms; +2.027ms] or [+2.414%; +7.164%]		 unstable
[-42.109op/s; +10.922op/s] or [-15.701%; +4.072%]		 18.082ms 29.658ms 252.594op/s 16.932ms 28.302ms 268.188op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.18 ms) : 1168, 1191
.   : milestone, 1180,
iast (3.215 ms) : 3169, 3261
.   : milestone, 3215,
iast_FULL (5.738 ms) : 5681, 5796
.   : milestone, 5738,
iast_GLOBAL (3.604 ms) : 3541, 3667
.   : milestone, 3604,
profiling (1.958 ms) : 1940, 1977
.   : milestone, 1958,
tracing (1.776 ms) : 1761, 1791
.   : milestone, 1776,
section candidate
no_agent (1.23 ms) : 1217, 1242
.   : milestone, 1230,
iast (3.177 ms) : 3138, 3216
.   : milestone, 3177,
iast_FULL (5.764 ms) : 5706, 5822
.   : milestone, 5764,
iast_GLOBAL (3.472 ms) : 3414, 3529
.   : milestone, 3472,
profiling (2.219 ms) : 2199, 2240
.   : milestone, 2219,
tracing (1.777 ms) : 1762, 1792
.   : milestone, 1777,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.18 ms [1.168 ms, 1.191 ms] -
iast 3.215 ms [3.169 ms, 3.261 ms] 2.035 ms (172.5%)
iast_FULL 5.738 ms [5.681 ms, 5.796 ms] 4.559 ms (386.4%)
iast_GLOBAL 3.604 ms [3.541 ms, 3.667 ms] 2.424 ms (205.5%)
profiling 1.958 ms [1.94 ms, 1.977 ms] 778.429 µs (66.0%)
tracing 1.776 ms [1.761 ms, 1.791 ms] 596.194 µs (50.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.23 ms [1.217 ms, 1.242 ms] -
iast 3.177 ms [3.138 ms, 3.216 ms] 1.947 ms (158.4%)
iast_FULL 5.764 ms [5.706 ms, 5.822 ms] 4.534 ms (368.8%)
iast_GLOBAL 3.472 ms [3.414 ms, 3.529 ms] 2.242 ms (182.3%)
profiling 2.219 ms [2.199 ms, 2.24 ms] 989.862 µs (80.5%)
tracing 1.777 ms [1.762 ms, 1.792 ms] 547.45 µs (44.5%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901
    dateFormat X
    axisFormat %s
section baseline
no_agent (18.028 ms) : 17846, 18210
.   : milestone, 18028,
appsec (18.482 ms) : 18294, 18671
.   : milestone, 18482,
code_origins (17.641 ms) : 17462, 17820
.   : milestone, 17641,
iast (17.396 ms) : 17222, 17570
.   : milestone, 17396,
profiling (18.95 ms) : 18759, 19141
.   : milestone, 18950,
tracing (17.779 ms) : 17606, 17953
.   : milestone, 17779,
section candidate
no_agent (18.078 ms) : 17888, 18268
.   : milestone, 18078,
appsec (18.478 ms) : 18292, 18663
.   : milestone, 18478,
code_origins (17.58 ms) : 17404, 17757
.   : milestone, 17580,
iast (18.48 ms) : 18292, 18667
.   : milestone, 18480,
profiling (18.692 ms) : 18503, 18880
.   : milestone, 18692,
tracing (17.513 ms) : 17341, 17684
.   : milestone, 17513,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.028 ms [17.846 ms, 18.21 ms] -
appsec 18.482 ms [18.294 ms, 18.671 ms] 453.839 µs (2.5%)
code_origins 17.641 ms [17.462 ms, 17.82 ms] -387.227 µs (-2.1%)
iast 17.396 ms [17.222 ms, 17.57 ms] -631.923 µs (-3.5%)
profiling 18.95 ms [18.759 ms, 19.141 ms] 921.755 µs (5.1%)
tracing 17.779 ms [17.606 ms, 17.953 ms] -248.878 µs (-1.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.078 ms [17.888 ms, 18.268 ms] -
appsec 18.478 ms [18.292 ms, 18.663 ms] 399.732 µs (2.2%)
code_origins 17.58 ms [17.404 ms, 17.757 ms] -497.751 µs (-2.8%)
iast 18.48 ms [18.292 ms, 18.667 ms] 401.746 µs (2.2%)
profiling 18.692 ms [18.503 ms, 18.88 ms] 613.809 µs (3.4%)
tracing 17.513 ms [17.341 ms, 17.684 ms] -565.293 µs (-3.1%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master bdu/total-tests
git_commit_date 1770829518 1770842148
git_commit_sha 63b528a fecd36e
release_version 1.60.0-SNAPSHOT~63b528a901 1.60.0-SNAPSHOT~fecd36eaa8
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1770844173 1770844173
ci_job_id 1419103738 1419103738
ci_pipeline_id 95942597 95942597
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-0l6cwtwp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-0l6cwtwp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.474 ms) : 1463, 1486
.   : milestone, 1474,
appsec (3.764 ms) : 3542, 3986
.   : milestone, 3764,
iast (2.26 ms) : 2192, 2329
.   : milestone, 2260,
iast_GLOBAL (2.302 ms) : 2233, 2372
.   : milestone, 2302,
profiling (2.092 ms) : 2037, 2147
.   : milestone, 2092,
tracing (2.064 ms) : 2011, 2118
.   : milestone, 2064,
section candidate
no_agent (1.478 ms) : 1466, 1489
.   : milestone, 1478,
appsec (3.782 ms) : 3559, 4004
.   : milestone, 3782,
iast (2.259 ms) : 2190, 2328
.   : milestone, 2259,
iast_GLOBAL (2.314 ms) : 2244, 2383
.   : milestone, 2314,
profiling (2.505 ms) : 2341, 2670
.   : milestone, 2505,
tracing (2.067 ms) : 2014, 2121
.   : milestone, 2067,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.474 ms [1.463 ms, 1.486 ms] -
appsec 3.764 ms [3.542 ms, 3.986 ms] 2.29 ms (155.3%)
iast 2.26 ms [2.192 ms, 2.329 ms] 786.124 µs (53.3%)
iast_GLOBAL 2.302 ms [2.233 ms, 2.372 ms] 828.13 µs (56.2%)
profiling 2.092 ms [2.037 ms, 2.147 ms] 617.708 µs (41.9%)
tracing 2.064 ms [2.011 ms, 2.118 ms] 590.157 µs (40.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.478 ms [1.466 ms, 1.489 ms] -
appsec 3.782 ms [3.559 ms, 4.004 ms] 2.304 ms (155.9%)
iast 2.259 ms [2.19 ms, 2.328 ms] 781.779 µs (52.9%)
iast_GLOBAL 2.314 ms [2.244 ms, 2.383 ms] 836.014 µs (56.6%)
profiling 2.505 ms [2.341 ms, 2.67 ms] 1.028 ms (69.6%)
tracing 2.067 ms [2.014 ms, 2.121 ms] 589.603 µs (39.9%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~fecd36eaa8, baseline=1.60.0-SNAPSHOT~63b528a901
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.505 s) : 15505000, 15505000
.   : milestone, 15505000,
appsec (14.778 s) : 14778000, 14778000
.   : milestone, 14778000,
iast (18.454 s) : 18454000, 18454000
.   : milestone, 18454000,
iast_GLOBAL (17.799 s) : 17799000, 17799000
.   : milestone, 17799000,
profiling (15.203 s) : 15203000, 15203000
.   : milestone, 15203000,
tracing (14.705 s) : 14705000, 14705000
.   : milestone, 14705000,
section candidate
no_agent (15.606 s) : 15606000, 15606000
.   : milestone, 15606000,
appsec (14.816 s) : 14816000, 14816000
.   : milestone, 14816000,
iast (18.096 s) : 18096000, 18096000
.   : milestone, 18096000,
iast_GLOBAL (17.856 s) : 17856000, 17856000
.   : milestone, 17856000,
profiling (15.068 s) : 15068000, 15068000
.   : milestone, 15068000,
tracing (14.77 s) : 14770000, 14770000
.   : milestone, 14770000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.505 s [15.505 s, 15.505 s] -
appsec 14.778 s [14.778 s, 14.778 s] -727.0 ms (-4.7%)
iast 18.454 s [18.454 s, 18.454 s] 2.949 s (19.0%)
iast_GLOBAL 17.799 s [17.799 s, 17.799 s] 2.294 s (14.8%)
profiling 15.203 s [15.203 s, 15.203 s] -302.0 ms (-1.9%)
tracing 14.705 s [14.705 s, 14.705 s] -800.0 ms (-5.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.606 s [15.606 s, 15.606 s] -
appsec 14.816 s [14.816 s, 14.816 s] -790.0 ms (-5.1%)
iast 18.096 s [18.096 s, 18.096 s] 2.49 s (16.0%)
iast_GLOBAL 17.856 s [17.856 s, 17.856 s] 2.25 s (14.4%)
profiling 15.068 s [15.068 s, 15.068 s] -538.0 ms (-3.4%)
tracing 14.77 s [14.77 s, 14.77 s] -836.0 ms (-5.4%)

@bric3 bric3 added the tag: ai generated Largely based on code generated by an AI or LLM label Feb 5, 2026
@bric3
Copy link
Copy Markdown
Contributor Author

bric3 commented Feb 5, 2026

FYI I tagged it tag: ai generated, but it was more pair programming than largely generated by AI.

@bric3 bric3 changed the title Compute ran tests in a pipeline Tests summary in pipeline and improved test tags to be used in Test Optimization producrt Feb 5, 2026
@bric3 bric3 changed the title Tests summary in pipeline and improved test tags to be used in Test Optimization producrt Tests summary in pipeline and improved test tags to be used in Test Optimization product Feb 5, 2026
PerfectSlayer
PerfectSlayer reviewed Feb 9, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@PerfectSlayer PerfectSlayer left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 thought: ‏That's very big shell script with complex jq query. Not sure what kind of feedback I can give as part of the review

If you feel confident about it, I can vet and approve but I struggle to review it.

@sarahchen6
Copy link
Copy Markdown
Contributor

sarahchen6 commented Feb 10, 2026

Same comment as Bruce above 😅 The log breakdowns and new JUnit queries seem promising though! An idea could be to also output the % change compared to number of tests run on master. The monitor that Charles created covers just master too.

@bric3
Copy link
Copy Markdown
Contributor Author

bric3 commented Feb 11, 2026

The monitor that Charles created covers just master too.

Yeah we discussed this monitor, it's really nice to have but doesn't report anything on PRs.

One of the thing the monitor won't do is the gate the PR if tests are not run.

@sarahchen6
Copy link
Copy Markdown
Contributor

sarahchen6 commented Feb 11, 2026

Should we be able to see the test failures (i.e. with dd-gitlab/test_inst: [21, 2/6] and dd-gitlab/test_inst: [21, 4/6]) in the aggregated_test_counts report? 🤔

@bric3 bric3 enabled auto-merge (squash) February 11, 2026 17:26
@bric3 bric3 disabled auto-merge February 11, 2026 17:29
@bric3 bric3 enabled auto-merge (squash) February 11, 2026 17:43
@bric3
Copy link
Copy Markdown
Contributor Author

bric3 commented Feb 11, 2026

Good catch, this is indeed something to look at. The issue is that the test didn't fail per se, it timed out and no junit report was written. Which this script uses.

> Task :dd-java-agent:instrumentation:java:java-lang:java-lang-21.0:test
Taking dumps after 1140 seconds delay for :dd-java-agent:instrumentation:java:java-lang:java-lang-21.0:test
Requesting stop of task ':dd-java-agent:instrumentation:java:java-lang:java-lang-21.0:test' as it has exceeded its configured timeout of 20m.

I opened a ticket on this gradle/gradle#36699, as I'm not sure Gradle supports it today.

Note this log line is coming from our custom DumpHangedTestPlugin

Taking dumps after 1140 seconds delay for :dd-java-agent:instrumentation:java:java-lang:java-lang-21.0:test

@bric3 bric3 merged commit c706f2b into master Feb 11, 2026
716 of 718 checks passed
@bric3 bric3 deleted the bdu/total-tests branch February 11, 2026 22:38
@github-actions github-actions Bot added this to the 1.60.0 milestone Feb 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@randomanderson randomanderson Awaiting requested review from randomanderson randomanderson is a code owner automatically assigned from DataDog/apm-release-platform

@sarahchen6 sarahchen6 Awaiting requested review from sarahchen6

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.60.0

Development

Successfully merging this pull request may close these issues.

3 participants

@bric3 @sarahchen6 @PerfectSlayer