Skip to content

Add basic Scala MUnit support#5781

Merged
nikita-tkachenko-datadog merged 3 commits into
masterfrom
nikita-tkachenko/basic-scala-munit-support
Aug 28, 2023
Merged

Add basic Scala MUnit support#5781
nikita-tkachenko-datadog merged 3 commits into
masterfrom
nikita-tkachenko/basic-scala-munit-support

Conversation

@nikita-tkachenko-datadog

@nikita-tkachenko-datadog nikita-tkachenko-datadog commented Aug 25, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Adds support for Scala MUnit testing framework to CI Visibility.

Motivation

There is a feature request.

Additional Notes

MUnit is built on top of JUnit 4.

In order to support it, the following things were done:

  • reworked test suite events mechanism: "custom" suite events implemented via bytecode instrumentation are now used only for JUnit < 4.13. Versions 4.13 and above now use "native" suite events that are available as part of the framework. The reason to change this was that the custom events were not working properly with custom runner implementations
  • reworked logic that detects if a Description contains direct children that are atomic test cases: it was not working properly with MUnit, causing all suites to be ignored
  • added a separate method to handle MUnit tests skipping, since there are some differences compared to the other frameworks (when skipping a test case, separate test start event is received; when skipping a suite, suite start and finished events are received)
  • updated logic that detects test framework name and version: it now takes MUnit into account
  • updated repository indexing logic to handle Scala classes

@nikita-tkachenko-datadog nikita-tkachenko-datadog added the comp: ci visibility Continuous Integration Visibility label Aug 25, 2023
@pr-commenter

pr-commenter Bot commented Aug 25, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~acb417b5b2 1.20.0-SNAPSHOT~a1dbe0de0b
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (989.452 ms) : 0, 989452
Total [baseline] (9.184 s) : 0, 9183839
Agent [candidate] (992.687 ms) : 0, 992687
Total [candidate] (9.221 s) : 0, 9220912
section appsec
Agent [baseline] (1.081 s) : 0, 1080993
Total [baseline] (9.293 s) : 0, 9293121
Agent [candidate] (1.072 s) : 0, 1072420
Total [candidate] (9.238 s) : 0, 9238382
section iast
Agent [baseline] (1.092 s) : 0, 1091656
Total [baseline] (9.401 s) : 0, 9401418
Agent [candidate] (1.09 s) : 0, 1090287
Total [candidate] (9.373 s) : 0, 9372904
section profiling
Agent [baseline] (1.157 s) : 0, 1156877
Total [baseline] (9.483 s) : 0, 9483416
Agent [candidate] (1.159 s) : 0, 1158920
Total [candidate] (9.438 s) : 0, 9437591
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 989.452 ms -
Agent appsec 1.081 s 91.541 ms (9.3%)
Agent iast 1.092 s 102.204 ms (10.3%)
Agent profiling 1.157 s 167.424 ms (16.9%)
Total tracing 9.184 s -
Total appsec 9.293 s 109.282 ms (1.2%)
Total iast 9.401 s 217.579 ms (2.4%)
Total profiling 9.483 s 299.577 ms (3.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 992.687 ms -
Agent appsec 1.072 s 79.733 ms (8.0%)
Agent iast 1.09 s 97.6 ms (9.8%)
Agent profiling 1.159 s 166.233 ms (16.7%)
Total tracing 9.221 s -
Total appsec 9.238 s 17.47 ms (0.2%)
Total iast 9.373 s 151.991 ms (1.6%)
Total profiling 9.438 s 216.679 ms (2.3%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (613.431 ms) : 0, 613431
BytebuddyAgent [candidate] (615.471 ms) : 0, 615471
GlobalTracer [baseline] (284.507 ms) : 0, 284507
GlobalTracer [candidate] (286.538 ms) : 0, 286538
AppSec [baseline] (50.344 ms) : 0, 50344
AppSec [candidate] (49.487 ms) : 0, 49487
Remote Config [baseline] (663.743 µs) : 0, 664
Remote Config [candidate] (655.246 µs) : 0, 655
Telemetry [baseline] (6.063 ms) : 0, 6063
Telemetry [candidate] (6.02 ms) : 0, 6020
section appsec
BytebuddyAgent [baseline] (617.547 ms) : 0, 617547
BytebuddyAgent [candidate] (612.27 ms) : 0, 612270
GlobalTracer [baseline] (286.561 ms) : 0, 286561
GlobalTracer [candidate] (284.547 ms) : 0, 284547
AppSec [baseline] (135.884 ms) : 0, 135884
AppSec [candidate] (134.866 ms) : 0, 134866
Remote Config [baseline] (728.261 µs) : 0, 728
Remote Config [candidate] (724.518 µs) : 0, 725
Telemetry [baseline] (5.796 ms) : 0, 5796
Telemetry [candidate] (5.725 ms) : 0, 5725
section iast
BytebuddyAgent [baseline] (719.259 ms) : 0, 719259
BytebuddyAgent [candidate] (718.331 ms) : 0, 718331
GlobalTracer [baseline] (270.537 ms) : 0, 270537
GlobalTracer [candidate] (270.431 ms) : 0, 270431
AppSec [baseline] (47.443 ms) : 0, 47443
AppSec [candidate] (47.159 ms) : 0, 47159
IAST [baseline] (13.879 ms) : 0, 13879
IAST [candidate] (13.904 ms) : 0, 13904
Remote Config [baseline] (553.835 µs) : 0, 554
Remote Config [candidate] (545.433 µs) : 0, 545
Telemetry [baseline] (5.672 ms) : 0, 5672
Telemetry [candidate] (5.658 ms) : 0, 5658
section profiling
BytebuddyAgent [baseline] (625.274 ms) : 0, 625274
BytebuddyAgent [candidate] (626.512 ms) : 0, 626512
GlobalTracer [baseline] (346.049 ms) : 0, 346049
GlobalTracer [candidate] (345.74 ms) : 0, 345740
AppSec [baseline] (48.934 ms) : 0, 48934
AppSec [candidate] (49.204 ms) : 0, 49204
Remote Config [baseline] (647.947 µs) : 0, 648
Remote Config [candidate] (649.442 µs) : 0, 649
Telemetry [baseline] (5.955 ms) : 0, 5955
Telemetry [candidate] (5.99 ms) : 0, 5990
ProfilingAgent [baseline] (77.156 ms) : 0, 77156
ProfilingAgent [candidate] (77.895 ms) : 0, 77895
Profiling [baseline] (77.181 ms) : 0, 77181
Profiling [candidate] (77.919 ms) : 0, 77919
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (988.66 ms) : 0, 988660
Total [baseline] (8.705 s) : 0, 8704950
Agent [candidate] (989.027 ms) : 0, 989027
Total [candidate] (8.629 s) : 0, 8628679
section appsec
Agent [baseline] (1.072 s) : 0, 1072296
Total [baseline] (8.768 s) : 0, 8767949
Agent [candidate] (1.08 s) : 0, 1080319
Total [candidate] (8.764 s) : 0, 8764150
section iast
Agent [baseline] (1.09 s) : 0, 1089522
Total [baseline] (9.142 s) : 0, 9142321
Agent [candidate] (1.091 s) : 0, 1090968
Total [candidate] (9.151 s) : 0, 9150941
section profiling
Agent [baseline] (1.157 s) : 0, 1156757
Total [baseline] (8.864 s) : 0, 8864446
Agent [candidate] (1.159 s) : 0, 1158809
Total [candidate] (8.916 s) : 0, 8915931
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 988.66 ms -
Agent appsec 1.072 s 83.635 ms (8.5%)
Agent iast 1.09 s 100.861 ms (10.2%)
Agent profiling 1.157 s 168.096 ms (17.0%)
Total tracing 8.705 s -
Total appsec 8.768 s 62.999 ms (0.7%)
Total iast 9.142 s 437.37 ms (5.0%)
Total profiling 8.864 s 159.495 ms (1.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 989.027 ms -
Agent appsec 1.08 s 91.292 ms (9.2%)
Agent iast 1.091 s 101.941 ms (10.3%)
Agent profiling 1.159 s 169.782 ms (17.2%)
Total tracing 8.629 s -
Total appsec 8.764 s 135.471 ms (1.6%)
Total iast 9.151 s 522.262 ms (6.1%)
Total profiling 8.916 s 287.252 ms (3.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (613.6 ms) : 0, 613600
BytebuddyAgent [candidate] (613.381 ms) : 0, 613381
GlobalTracer [baseline] (284.328 ms) : 0, 284328
GlobalTracer [candidate] (284.568 ms) : 0, 284568
AppSec [baseline] (49.734 ms) : 0, 49734
AppSec [candidate] (49.959 ms) : 0, 49959
Remote Config [baseline] (654.296 µs) : 0, 654
Remote Config [candidate] (664.676 µs) : 0, 665
Telemetry [baseline] (5.965 ms) : 0, 5965
Telemetry [candidate] (6.004 ms) : 0, 6004
section appsec
BytebuddyAgent [baseline] (612.024 ms) : 0, 612024
BytebuddyAgent [candidate] (617.17 ms) : 0, 617170
GlobalTracer [baseline] (284.008 ms) : 0, 284008
GlobalTracer [candidate] (286.645 ms) : 0, 286645
AppSec [baseline] (135.575 ms) : 0, 135575
AppSec [candidate] (135.429 ms) : 0, 135429
Remote Config [baseline] (716.028 µs) : 0, 716
Remote Config [candidate] (724.658 µs) : 0, 725
Telemetry [baseline] (5.721 ms) : 0, 5721
Telemetry [candidate] (5.785 ms) : 0, 5785
section iast
BytebuddyAgent [baseline] (718.222 ms) : 0, 718222
BytebuddyAgent [candidate] (718.484 ms) : 0, 718484
GlobalTracer [baseline] (269.698 ms) : 0, 269698
GlobalTracer [candidate] (270.915 ms) : 0, 270915
AppSec [baseline] (47.361 ms) : 0, 47361
AppSec [candidate] (47.254 ms) : 0, 47254
IAST [baseline] (13.715 ms) : 0, 13715
IAST [candidate] (13.74 ms) : 0, 13740
Remote Config [baseline] (548.602 µs) : 0, 549
Remote Config [candidate] (540.558 µs) : 0, 541
Telemetry [baseline] (5.706 ms) : 0, 5706
Telemetry [candidate] (5.66 ms) : 0, 5660
section profiling
BytebuddyAgent [baseline] (625.904 ms) : 0, 625904
BytebuddyAgent [candidate] (626.176 ms) : 0, 626176
GlobalTracer [baseline] (345.611 ms) : 0, 345611
GlobalTracer [candidate] (347.942 ms) : 0, 347942
AppSec [baseline] (48.818 ms) : 0, 48818
AppSec [candidate] (49.487 ms) : 0, 49487
Remote Config [baseline] (646.681 µs) : 0, 647
Remote Config [candidate] (650.461 µs) : 0, 650
Telemetry [baseline] (5.961 ms) : 0, 5961
Telemetry [candidate] (6.024 ms) : 0, 6024
ProfilingAgent [baseline] (76.954 ms) : 0, 76954
ProfilingAgent [candidate] (75.713 ms) : 0, 75713
Profiling [baseline] (76.979 ms) : 0, 76979
Profiling [candidate] (75.737 ms) : 0, 75737
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~acb417b5b2 1.20.0-SNAPSHOT~a1dbe0de0b
config baseline candidate
end_time 2023-08-25T12:07:06 2023-08-25T12:25:10
start_time 2023-08-25T12:06:48 2023-08-25T12:24:52
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.33 ms) : 1311, 1349
.   : milestone, 1330,
appsec (1.657 ms) : 1632, 1681
.   : milestone, 1657,
iast (1.471 ms) : 1447, 1495
.   : milestone, 1471,
profiling (1.455 ms) : 1431, 1480
.   : milestone, 1455,
tracing (1.454 ms) : 1429, 1479
.   : milestone, 1454,
section candidate
no_agent (1.33 ms) : 1312, 1349
.   : milestone, 1330,
appsec (1.656 ms) : 1632, 1681
.   : milestone, 1656,
iast (1.431 ms) : 1407, 1456
.   : milestone, 1431,
profiling (1.45 ms) : 1425, 1476
.   : milestone, 1450,
tracing (1.427 ms) : 1403, 1452
.   : milestone, 1427,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.33 ms [1.311 ms, 1.349 ms] -
appsec 1.657 ms [1.632 ms, 1.681 ms] 326.792 µs (24.6%)
iast 1.471 ms [1.447 ms, 1.495 ms] 141.097 µs (10.6%)
profiling 1.455 ms [1.431 ms, 1.48 ms] 125.338 µs (9.4%)
tracing 1.454 ms [1.429 ms, 1.479 ms] 123.657 µs (9.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.33 ms [1.312 ms, 1.349 ms] -
appsec 1.656 ms [1.632 ms, 1.681 ms] 326.021 µs (24.5%)
iast 1.431 ms [1.407 ms, 1.456 ms] 101.231 µs (7.6%)
profiling 1.45 ms [1.425 ms, 1.476 ms] 120.249 µs (9.0%)
tracing 1.427 ms [1.403 ms, 1.452 ms] 97.092 µs (7.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~a1dbe0de0b, baseline=1.20.0-SNAPSHOT~acb417b5b2
    dateFormat X
    axisFormat %s
section baseline
no_agent (365.246 µs) : 346, 385
.   : milestone, 365,
appsec (662.345 µs) : 642, 683
.   : milestone, 662,
iast (447.328 µs) : 427, 468
.   : milestone, 447,
iast_FULL (519.646 µs) : 499, 540
.   : milestone, 520,
iast_INACTIVE (431.921 µs) : 411, 453
.   : milestone, 432,
profiling (433.625 µs) : 413, 454
.   : milestone, 434,
tracing (428.63 µs) : 408, 449
.   : milestone, 429,
section candidate
no_agent (364.529 µs) : 344, 385
.   : milestone, 365,
appsec (667.714 µs) : 646, 689
.   : milestone, 668,
iast (457.86 µs) : 437, 479
.   : milestone, 458,
iast_FULL (527.422 µs) : 507, 548
.   : milestone, 527,
iast_INACTIVE (432.01 µs) : 410, 454
.   : milestone, 432,
profiling (442.697 µs) : 422, 464
.   : milestone, 443,
tracing (435.142 µs) : 414, 456
.   : milestone, 435,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 365.246 µs [345.625 µs, 384.868 µs] -
appsec 662.345 µs [641.979 µs, 682.711 µs] 297.099 µs (81.3%)
iast 447.328 µs [426.788 µs, 467.869 µs] 82.082 µs (22.5%)
iast_FULL 519.646 µs [498.902 µs, 540.39 µs] 154.399 µs (42.3%)
iast_INACTIVE 431.921 µs [410.815 µs, 453.026 µs] 66.675 µs (18.3%)
profiling 433.625 µs [412.852 µs, 454.398 µs] 68.378 µs (18.7%)
tracing 428.63 µs [407.852 µs, 449.407 µs] 63.383 µs (17.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.529 µs [343.727 µs, 385.331 µs] -
appsec 667.714 µs [646.222 µs, 689.206 µs] 303.185 µs (83.2%)
iast 457.86 µs [437.048 µs, 478.673 µs] 93.331 µs (25.6%)
iast_FULL 527.422 µs [506.671 µs, 548.173 µs] 162.893 µs (44.7%)
iast_INACTIVE 432.01 µs [410.366 µs, 453.653 µs] 67.48 µs (18.5%)
profiling 442.697 µs [421.875 µs, 463.52 µs] 78.168 µs (21.4%)
tracing 435.142 µs [413.97 µs, 456.314 µs] 70.613 µs (19.4%)

@nikita-tkachenko-datadog nikita-tkachenko-datadog mentioned this pull request Aug 25, 2023
Closed
@nikita-tkachenko-datadog
nikita-tkachenko-datadog marked this pull request as ready for review August 25, 2023 13:10
@nikita-tkachenko-datadog
nikita-tkachenko-datadog requested a review from a team as a code owner August 25, 2023 13:10
@nikita-tkachenko-datadog
nikita-tkachenko-datadog merged commit 1bdf8d3 into master Aug 28, 2023
@nikita-tkachenko-datadog
nikita-tkachenko-datadog deleted the nikita-tkachenko/basic-scala-munit-support branch August 28, 2023 16:03
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 28, 2023
@nikita-tkachenko-datadog nikita-tkachenko-datadog linked an issue Aug 31, 2023 that may be closed by this pull request
Support MUnit in CI Tests #5760
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@juan-fernandez juan-fernandez juan-fernandez approved these changes

Assignees

No one assigned

Labels

comp: ci visibility Continuous Integration Visibility

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

Support MUnit in CI Tests

2 participants

@nikita-tkachenko-datadog @juan-fernandez