Skip to content

Parallelize system-tests#7954

Merged
smola merged 1 commit intomasterfrom
smola/parallelize-system-tests
Nov 14, 2024
Merged

Parallelize system-tests#7954
smola merged 1 commit intomasterfrom
smola/parallelize-system-tests

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Nov 14, 2024 •
edited
Loading

What Does This Do

Parallelize system-tests at scenario level. We just use circleci tests for splitting, which will be quite naive, but might just do the job at the moment.

Motivation

I'll be expanding the number of scenarios that are run, and running them linearly will increase the wall clock time for the full pipeline. Parallelization here is quite reasonable since there's just ~2min of the job that is redundant, and these are medium size runners anyway.

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Nov 14, 2024
@smola smola force-pushed the smola/parallelize-system-tests branch from 960d9cd to 76ac62a Compare November 14, 2024 13:19
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Nov 14, 2024

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/parallelize-system-tests
git_commit_date 1731578914 1731590321
git_commit_sha e7571fd 76ac62a
release_version 1.43.0-SNAPSHOT~e7571fd2ee 1.43.0-SNAPSHOT~76ac62a6a7
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1731592672 1731592672
ci_job_id 706908983 706908983
ci_pipeline_id 49009338 49009338
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 9 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.09 s) : 0, 1090243
Total [baseline] (8.585 s) : 0, 8584947
Agent [candidate] (1.079 s) : 0, 1078737
Total [candidate] (8.597 s) : 0, 8597161
section iast
Agent [baseline] (1.206 s) : 0, 1206197
Total [baseline] (9.122 s) : 0, 9122063
Agent [candidate] (1.21 s) : 0, 1209623
Total [candidate] (9.174 s) : 0, 9173508
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.206 s) : 0, 1206266
Total [baseline] (9.097 s) : 0, 9097043
Agent [candidate] (1.206 s) : 0, 1205926
Total [candidate] (9.108 s) : 0, 9107843
section iast_TELEMETRY_OFF
Agent [baseline] (1.206 s) : 0, 1205833
Total [baseline] (9.131 s) : 0, 9131205
Agent [candidate] (1.203 s) : 0, 1202702
Total [candidate] (9.129 s) : 0, 9128608
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.09 s -
Agent iast 1.206 s 115.954 ms (10.6%)
Agent iast_HARDCODED_SECRET_DISABLED 1.206 s 116.023 ms (10.6%)
Agent iast_TELEMETRY_OFF 1.206 s 115.59 ms (10.6%)
Total tracing 8.585 s -
Total iast 9.122 s 537.115 ms (6.3%)
Total iast_HARDCODED_SECRET_DISABLED 9.097 s 512.096 ms (6.0%)
Total iast_TELEMETRY_OFF 9.131 s 546.257 ms (6.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.079 s -
Agent iast 1.21 s 130.886 ms (12.1%)
Agent iast_HARDCODED_SECRET_DISABLED 1.206 s 127.189 ms (11.8%)
Agent iast_TELEMETRY_OFF 1.203 s 123.964 ms (11.5%)
Total tracing 8.597 s -
Total iast 9.174 s 576.347 ms (6.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.108 s 510.682 ms (5.9%)
Total iast_TELEMETRY_OFF 9.129 s 531.447 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (691.722 ms) : 0, 691722
BytebuddyAgent [candidate] (685.658 ms) : 0, 685658
GlobalTracer [baseline] (317.391 ms) : 0, 317391
GlobalTracer [candidate] (314.857 ms) : 0, 314857
AppSec [baseline] (54.75 ms) : 0, 54750
AppSec [candidate] (53.914 ms) : 0, 53914
Remote Config [baseline] (691.952 µs) : 0, 692
Remote Config [candidate] (680.454 µs) : 0, 680
Telemetry [baseline] (11.876 ms) : 0, 11876
Telemetry [candidate] (9.951 ms) : 0, 9951
section iast
BytebuddyAgent [baseline] (801.975 ms) : 0, 801975
BytebuddyAgent [candidate] (804.369 ms) : 0, 804369
GlobalTracer [baseline] (304.486 ms) : 0, 304486
GlobalTracer [candidate] (305.665 ms) : 0, 305665
AppSec [baseline] (56.595 ms) : 0, 56595
AppSec [candidate] (57.48 ms) : 0, 57480
IAST [baseline] (21.429 ms) : 0, 21429
IAST [candidate] (20.327 ms) : 0, 20327
Remote Config [baseline] (614.5 µs) : 0, 614
Remote Config [candidate] (626.275 µs) : 0, 626
Telemetry [baseline] (7.466 ms) : 0, 7466
Telemetry [candidate] (7.427 ms) : 0, 7427
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (802.134 ms) : 0, 802134
BytebuddyAgent [candidate] (801.391 ms) : 0, 801391
GlobalTracer [baseline] (304.439 ms) : 0, 304439
GlobalTracer [candidate] (304.587 ms) : 0, 304587
AppSec [baseline] (57.424 ms) : 0, 57424
AppSec [candidate] (56.689 ms) : 0, 56689
IAST [baseline] (20.559 ms) : 0, 20559
IAST [candidate] (21.493 ms) : 0, 21493
Remote Config [baseline] (605.409 µs) : 0, 605
Remote Config [candidate] (609.888 µs) : 0, 610
Telemetry [baseline] (7.457 ms) : 0, 7457
Telemetry [candidate] (7.489 ms) : 0, 7489
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (801.593 ms) : 0, 801593
BytebuddyAgent [candidate] (798.795 ms) : 0, 798795
GlobalTracer [baseline] (304.555 ms) : 0, 304555
GlobalTracer [candidate] (304.002 ms) : 0, 304002
AppSec [baseline] (57.082 ms) : 0, 57082
AppSec [candidate] (57.12 ms) : 0, 57120
IAST [baseline] (20.974 ms) : 0, 20974
IAST [candidate] (21.106 ms) : 0, 21106
Remote Config [baseline] (608.772 µs) : 0, 609
Remote Config [candidate] (608.439 µs) : 0, 608
Telemetry [baseline] (7.343 ms) : 0, 7343
Telemetry [candidate] (7.399 ms) : 0, 7399
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.085 s) : 0, 1085117
Total [baseline] (10.428 s) : 0, 10427954
Agent [candidate] (1.081 s) : 0, 1080804
Total [candidate] (10.4 s) : 0, 10399584
section appsec
Agent [baseline] (1.215 s) : 0, 1214895
Total [baseline] (10.623 s) : 0, 10623406
Agent [candidate] (1.217 s) : 0, 1216816
Total [candidate] (10.674 s) : 0, 10673610
section iast
Agent [baseline] (1.217 s) : 0, 1217383
Total [baseline] (10.959 s) : 0, 10959303
Agent [candidate] (1.207 s) : 0, 1206908
Total [candidate] (10.9 s) : 0, 10900343
section profiling
Agent [baseline] (1.277 s) : 0, 1276699
Total [baseline] (10.696 s) : 0, 10696233
Agent [candidate] (1.289 s) : 0, 1288725
Total [candidate] (10.859 s) : 0, 10858986
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.085 s -
Agent appsec 1.215 s 129.778 ms (12.0%)
Agent iast 1.217 s 132.266 ms (12.2%)
Agent profiling 1.277 s 191.582 ms (17.7%)
Total tracing 10.428 s -
Total appsec 10.623 s 195.452 ms (1.9%)
Total iast 10.959 s 531.349 ms (5.1%)
Total profiling 10.696 s 268.279 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.081 s -
Agent appsec 1.217 s 136.011 ms (12.6%)
Agent iast 1.207 s 126.104 ms (11.7%)
Agent profiling 1.289 s 207.921 ms (19.2%)
Total tracing 10.4 s -
Total appsec 10.674 s 274.026 ms (2.6%)
Total iast 10.9 s 500.759 ms (4.8%)
Total profiling 10.859 s 459.402 ms (4.4%) 
gantt
    title petclinic - break down per module: candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.893 ms) : 0, 690893
BytebuddyAgent [candidate] (685.884 ms) : 0, 685884
GlobalTracer [baseline] (316.888 ms) : 0, 316888
GlobalTracer [candidate] (314.954 ms) : 0, 314954
AppSec [baseline] (54.411 ms) : 0, 54411
AppSec [candidate] (54.178 ms) : 0, 54178
Remote Config [baseline] (678.826 µs) : 0, 679
Remote Config [candidate] (678.062 µs) : 0, 678
Telemetry [baseline] (8.513 ms) : 0, 8513
Telemetry [candidate] (11.482 ms) : 0, 11482
section appsec
BytebuddyAgent [baseline] (703.871 ms) : 0, 703871
BytebuddyAgent [candidate] (704.774 ms) : 0, 704774
GlobalTracer [baseline] (312.801 ms) : 0, 312801
GlobalTracer [candidate] (313.41 ms) : 0, 313410
AppSec [baseline] (165.419 ms) : 0, 165419
AppSec [candidate] (166.718 ms) : 0, 166718
IAST [baseline] (20.148 ms) : 0, 20148
IAST [candidate] (20.313 ms) : 0, 20313
Remote Config [baseline] (630.506 µs) : 0, 631
Remote Config [candidate] (641.247 µs) : 0, 641
Telemetry [baseline] (8.453 ms) : 0, 8453
Telemetry [candidate] (7.464 ms) : 0, 7464
section iast
BytebuddyAgent [baseline] (810.725 ms) : 0, 810725
BytebuddyAgent [candidate] (802.755 ms) : 0, 802755
GlobalTracer [baseline] (306.616 ms) : 0, 306616
GlobalTracer [candidate] (304.809 ms) : 0, 304809
AppSec [baseline] (57.016 ms) : 0, 57016
AppSec [candidate] (56.336 ms) : 0, 56336
IAST [baseline] (21.237 ms) : 0, 21237
IAST [candidate] (21.293 ms) : 0, 21293
Remote Config [baseline] (612.004 µs) : 0, 612
Remote Config [candidate] (606.666 µs) : 0, 607
Telemetry [baseline] (7.396 ms) : 0, 7396
Telemetry [candidate] (7.436 ms) : 0, 7436
section profiling
BytebuddyAgent [baseline] (679.73 ms) : 0, 679730
BytebuddyAgent [candidate] (685.105 ms) : 0, 685105
GlobalTracer [baseline] (398.694 ms) : 0, 398694
GlobalTracer [candidate] (403.871 ms) : 0, 403871
AppSec [baseline] (54.613 ms) : 0, 54613
AppSec [candidate] (55.008 ms) : 0, 55008
Remote Config [baseline] (671.312 µs) : 0, 671
Remote Config [candidate] (672.765 µs) : 0, 673
Telemetry [baseline] (13.339 ms) : 0, 13339
Telemetry [candidate] (11.947 ms) : 0, 11947
ProfilingAgent [baseline] (90.824 ms) : 0, 90824
ProfilingAgent [candidate] (93.04 ms) : 0, 93040
Profiling [baseline] (90.848 ms) : 0, 90848
Profiling [candidate] (93.063 ms) : 0, 93063
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-11-14T13:28:12 2024-11-14T13:35:07
git_branch master smola/parallelize-system-tests
git_commit_date 1731578914 1731590321
git_commit_sha e7571fd 76ac62a
release_version 1.43.0-SNAPSHOT~e7571fd2ee 1.43.0-SNAPSHOT~76ac62a6a7
start_time 2024-11-14T13:27:58 2024-11-14T13:34:54
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1731591657 1731591657
ci_job_id 706908986 706908986
ci_pipeline_id 49009338 49009338
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee
    dateFormat X
    axisFormat %s
section baseline
no_agent (376.322 µs) : 356, 397
.   : milestone, 376,
iast (487.099 µs) : 466, 508
.   : milestone, 487,
iast_FULL (645.193 µs) : 624, 666
.   : milestone, 645,
iast_GLOBAL (518.309 µs) : 496, 540
.   : milestone, 518,
iast_HARDCODED_SECRET_DISABLED (486.424 µs) : 465, 508
.   : milestone, 486,
iast_INACTIVE (448.806 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (478.186 µs) : 457, 500
.   : milestone, 478,
tracing (446.615 µs) : 426, 467
.   : milestone, 447,
section candidate
no_agent (370.462 µs) : 351, 390
.   : milestone, 370,
iast (496.053 µs) : 475, 517
.   : milestone, 496,
iast_FULL (643.887 µs) : 623, 665
.   : milestone, 644,
iast_GLOBAL (531.935 µs) : 510, 554
.   : milestone, 532,
iast_HARDCODED_SECRET_DISABLED (485.846 µs) : 465, 507
.   : milestone, 486,
iast_INACTIVE (449.644 µs) : 429, 470
.   : milestone, 450,
iast_TELEMETRY_OFF (476.756 µs) : 455, 498
.   : milestone, 477,
tracing (436.373 µs) : 416, 456
.   : milestone, 436,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 376.322 µs [355.506 µs, 397.137 µs] -
iast 487.099 µs [465.79 µs, 508.408 µs] 110.778 µs (29.4%)
iast_FULL 645.193 µs [624.086 µs, 666.3 µs] 268.871 µs (71.4%)
iast_GLOBAL 518.309 µs [496.322 µs, 540.296 µs] 141.988 µs (37.7%)
iast_HARDCODED_SECRET_DISABLED 486.424 µs [465.017 µs, 507.831 µs] 110.102 µs (29.3%)
iast_INACTIVE 448.806 µs [427.78 µs, 469.832 µs] 72.484 µs (19.3%)
iast_TELEMETRY_OFF 478.186 µs [456.546 µs, 499.827 µs] 101.865 µs (27.1%)
tracing 446.615 µs [426.307 µs, 466.924 µs] 70.294 µs (18.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 370.462 µs [350.952 µs, 389.973 µs] -
iast 496.053 µs [474.659 µs, 517.447 µs] 125.591 µs (33.9%)
iast_FULL 643.887 µs [622.536 µs, 665.238 µs] 273.424 µs (73.8%)
iast_GLOBAL 531.935 µs [509.599 µs, 554.271 µs] 161.472 µs (43.6%)
iast_HARDCODED_SECRET_DISABLED 485.846 µs [464.502 µs, 507.189 µs] 115.383 µs (31.1%)
iast_INACTIVE 449.644 µs [428.794 µs, 470.493 µs] 79.182 µs (21.4%)
iast_TELEMETRY_OFF 476.756 µs [455.142 µs, 498.37 µs] 106.294 µs (28.7%)
tracing 436.373 µs [416.479 µs, 456.267 µs] 65.91 µs (17.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.345 ms) : 1325, 1364
.   : milestone, 1345,
appsec (1.728 ms) : 1705, 1752
.   : milestone, 1728,
appsec_no_iast (1.743 ms) : 1719, 1766
.   : milestone, 1743,
iast (1.503 ms) : 1480, 1525
.   : milestone, 1503,
profiling (1.523 ms) : 1498, 1547
.   : milestone, 1523,
tracing (1.466 ms) : 1440, 1491
.   : milestone, 1466,
section candidate
no_agent (1.333 ms) : 1313, 1353
.   : milestone, 1333,
appsec (1.743 ms) : 1719, 1768
.   : milestone, 1743,
appsec_no_iast (1.749 ms) : 1721, 1776
.   : milestone, 1749,
iast (1.48 ms) : 1457, 1502
.   : milestone, 1480,
profiling (1.489 ms) : 1466, 1512
.   : milestone, 1489,
tracing (1.472 ms) : 1447, 1497
.   : milestone, 1472,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.345 ms [1.325 ms, 1.364 ms] -
appsec 1.728 ms [1.705 ms, 1.752 ms] 383.655 µs (28.5%)
appsec_no_iast 1.743 ms [1.719 ms, 1.766 ms] 397.796 µs (29.6%)
iast 1.503 ms [1.48 ms, 1.525 ms] 157.954 µs (11.7%)
profiling 1.523 ms [1.498 ms, 1.547 ms] 177.951 µs (13.2%)
tracing 1.466 ms [1.44 ms, 1.491 ms] 120.89 µs (9.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.333 ms [1.313 ms, 1.353 ms] -
appsec 1.743 ms [1.719 ms, 1.768 ms] 409.968 µs (30.7%)
appsec_no_iast 1.749 ms [1.721 ms, 1.776 ms] 415.613 µs (31.2%)
iast 1.48 ms [1.457 ms, 1.502 ms] 146.252 µs (11.0%)
profiling 1.489 ms [1.466 ms, 1.512 ms] 155.828 µs (11.7%)
tracing 1.472 ms [1.447 ms, 1.497 ms] 138.944 µs (10.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/parallelize-system-tests
git_commit_date 1731578914 1731590321
git_commit_sha e7571fd 76ac62a
release_version 1.43.0-SNAPSHOT~e7571fd2ee 1.43.0-SNAPSHOT~76ac62a6a7
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1731592297 1731592297
ci_job_id 706908988 706908988
ci_pipeline_id 49009338 49009338
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.463 ms) : 1452, 1475
.   : milestone, 1463,
appsec (2.325 ms) : 2283, 2366
.   : milestone, 2325,
iast (2.057 ms) : 2006, 2109
.   : milestone, 2057,
iast_GLOBAL (2.098 ms) : 2046, 2150
.   : milestone, 2098,
profiling (1.947 ms) : 1904, 1991
.   : milestone, 1947,
tracing (1.908 ms) : 1868, 1947
.   : milestone, 1908,
section candidate
no_agent (1.461 ms) : 1450, 1473
.   : milestone, 1461,
appsec (2.308 ms) : 2267, 2348
.   : milestone, 2308,
iast (2.06 ms) : 2008, 2112
.   : milestone, 2060,
iast_GLOBAL (2.109 ms) : 2057, 2161
.   : milestone, 2109,
profiling (1.931 ms) : 1888, 1973
.   : milestone, 1931,
tracing (1.904 ms) : 1864, 1943
.   : milestone, 1904,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.463 ms [1.452 ms, 1.475 ms] -
appsec 2.325 ms [2.283 ms, 2.366 ms] 861.249 µs (58.9%)
iast 2.057 ms [2.006 ms, 2.109 ms] 593.989 µs (40.6%)
iast_GLOBAL 2.098 ms [2.046 ms, 2.15 ms] 634.323 µs (43.3%)
profiling 1.947 ms [1.904 ms, 1.991 ms] 484.069 µs (33.1%)
tracing 1.908 ms [1.868 ms, 1.947 ms] 444.307 µs (30.4%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.461 ms [1.45 ms, 1.473 ms] -
appsec 2.308 ms [2.267 ms, 2.348 ms] 846.432 µs (57.9%)
iast 2.06 ms [2.008 ms, 2.112 ms] 598.72 µs (41.0%)
iast_GLOBAL 2.109 ms [2.057 ms, 2.161 ms] 647.911 µs (44.3%)
profiling 1.931 ms [1.888 ms, 1.973 ms] 469.216 µs (32.1%)
tracing 1.904 ms [1.864 ms, 1.943 ms] 442.333 µs (30.3%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.43.0-SNAPSHOT~76ac62a6a7, baseline=1.43.0-SNAPSHOT~e7571fd2ee
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.102 s) : 15102000, 15102000
.   : milestone, 15102000,
appsec (15.24 s) : 15240000, 15240000
.   : milestone, 15240000,
iast (18.84 s) : 18840000, 18840000
.   : milestone, 18840000,
iast_GLOBAL (18.136 s) : 18136000, 18136000
.   : milestone, 18136000,
profiling (14.887 s) : 14887000, 14887000
.   : milestone, 14887000,
tracing (14.934 s) : 14934000, 14934000
.   : milestone, 14934000,
section candidate
no_agent (15.596 s) : 15596000, 15596000
.   : milestone, 15596000,
appsec (15.125 s) : 15125000, 15125000
.   : milestone, 15125000,
iast (18.487 s) : 18487000, 18487000
.   : milestone, 18487000,
iast_GLOBAL (18.113 s) : 18113000, 18113000
.   : milestone, 18113000,
profiling (15.012 s) : 15012000, 15012000
.   : milestone, 15012000,
tracing (15.228 s) : 15228000, 15228000
.   : milestone, 15228000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.102 s [15.102 s, 15.102 s] -
appsec 15.24 s [15.24 s, 15.24 s] 138.0 ms (0.9%)
iast 18.84 s [18.84 s, 18.84 s] 3.738 s (24.8%)
iast_GLOBAL 18.136 s [18.136 s, 18.136 s] 3.034 s (20.1%)
profiling 14.887 s [14.887 s, 14.887 s] -215.0 ms (-1.4%)
tracing 14.934 s [14.934 s, 14.934 s] -168.0 ms (-1.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.596 s [15.596 s, 15.596 s] -
appsec 15.125 s [15.125 s, 15.125 s] -471.0 ms (-3.0%)
iast 18.487 s [18.487 s, 18.487 s] 2.891 s (18.5%)
iast_GLOBAL 18.113 s [18.113 s, 18.113 s] 2.517 s (16.1%)
profiling 15.012 s [15.012 s, 15.012 s] -584.0 ms (-3.7%)
tracing 15.228 s [15.228 s, 15.228 s] -368.0 ms (-2.4%)

@smola smola marked this pull request as ready for review November 14, 2024 14:28
@smola smola requested a review from a team as a code owner November 14, 2024 14:28
@smola smola requested a review from dougqh November 14, 2024 14:28
@smola smola merged commit 105c9dc into master Nov 14, 2024
@smola smola deleted the smola/parallelize-system-tests branch November 14, 2024 16:31
@github-actions github-actions Bot added this to the 1.43.0 milestone Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcculls mcculls mcculls approved these changes

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.43.0

Development

Successfully merging this pull request may close these issues.

2 participants

@smola @mcculls