Skip to content

Check logs for errors at smoke tests cleanup#8111

Merged
smola merged 5 commits intomasterfrom
smola/smoke-test-log-errors
Dec 20, 2024
Merged

Check logs for errors at smoke tests cleanup#8111
smola merged 5 commits intomasterfrom
smola/smoke-test-log-errors

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Dec 18, 2024 •
edited
Loading

What Does This Do

  • Add boolean isErrorLog(String line) defining what is considered an error log for a given smoke test. Most smoke tests can just rely on the default. This PR includes several examples on how to extend it, both to add additional errors, or to exclude some errors.
  • Add void assertNoErrorLogs(final Closure<Boolean> errorFilter), which can be used to assert there are no error logs. Most tests should not use directly, and when they do, they should do it only after the test process has finished (otherwise, the test may be flaky).
  • Call assertNoErrorLogs on cleanupSpec for every smoke test. Most suites were not checking error logs.
  • Use processTestLogLines when we need to wait for a log line that is supposed to be printed. This should be less flaky than assertNoErrorLogs in the case where the test process has not ended.
  • Remove checkLogPostExit, which was superseded by the above methods.

Motivation

We have various flaky tests on smoke tests, and sometimes non-flaky failures that are hard to diagnose, that show up in errors all just as:

Condition not satisfied:

!logHasErrors
||
|true
false

This means there was an error printed to logs, but to find which one we need to download reports.tar and dig into the logs.

After this PR, they look like this:

Condition not satisfied:

errorLogs.isEmpty()
|         |
|         false
[16:39:08,836 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@5:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]], 16:39:08,984 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@15:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]]

Test application log contains 2 errors:
1: 16:39:08,836 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@5:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]
2: 16:39:08,984 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@15:21 - no applicable action for [immediateFlush], current pattern is [[configuration][appender][immediateFlush]]


	at datadog.smoketest.ProcessManager.assertNoErrorLogs(ProcessManager.groovy:294)
	at datadog.smoketest.ProcessManager.assertNoErrorLogs(ProcessManager.groovy:259)
	at datadog.smoketest.AbstractSmokeTest.cleanupSpec(AbstractSmokeTest.groovy:246)

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Dec 18, 2024
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Dec 18, 2024 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/smoke-test-log-errors
git_commit_date 1734636365 1734683196
git_commit_sha 88c493c 66897ba
release_version 1.45.0-SNAPSHOT~88c493cc76 1.45.0-SNAPSHOT~66897bacec
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1734685628 1734685628
ci_job_id 746725581 746725581
ci_pipeline_id 51638544 51638544
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 5 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.108 s) : 0, 1108194
Total [baseline] (8.668 s) : 0, 8667806
Agent [candidate] (1.108 s) : 0, 1108110
Total [candidate] (8.671 s) : 0, 8671110
section iast
Agent [baseline] (1.25 s) : 0, 1250141
Total [baseline] (9.252 s) : 0, 9251652
Agent [candidate] (1.241 s) : 0, 1240754
Total [candidate] (9.276 s) : 0, 9275654
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.241 s) : 0, 1241226
Total [baseline] (9.262 s) : 0, 9261918
Agent [candidate] (1.238 s) : 0, 1237733
Total [candidate] (9.303 s) : 0, 9302965
section iast_TELEMETRY_OFF
Agent [baseline] (1.237 s) : 0, 1236917
Total [baseline] (9.24 s) : 0, 9239535
Agent [candidate] (1.234 s) : 0, 1233776
Total [candidate] (9.213 s) : 0, 9213404
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.108 s -
Agent iast 1.25 s 141.947 ms (12.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.241 s 133.032 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.237 s 128.722 ms (11.6%)
Total tracing 8.668 s -
Total iast 9.252 s 583.846 ms (6.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.262 s 594.111 ms (6.9%)
Total iast_TELEMETRY_OFF 9.24 s 571.729 ms (6.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.108 s -
Agent iast 1.241 s 132.644 ms (12.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.238 s 129.623 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.234 s 125.666 ms (11.3%)
Total tracing 8.671 s -
Total iast 9.276 s 604.545 ms (7.0%)
Total iast_HARDCODED_SECRET_DISABLED 9.303 s 631.855 ms (7.3%)
Total iast_TELEMETRY_OFF 9.213 s 542.294 ms (6.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (710.699 ms) : 0, 710699
BytebuddyAgent [candidate] (710.942 ms) : 0, 710942
GlobalTracer [baseline] (318.058 ms) : 0, 318058
GlobalTracer [candidate] (318.066 ms) : 0, 318066
AppSec [baseline] (55.089 ms) : 0, 55089
AppSec [candidate] (54.72 ms) : 0, 54720
Remote Config [baseline] (664.651 µs) : 0, 665
Remote Config [candidate] (680.635 µs) : 0, 681
Telemetry [baseline] (8.864 ms) : 0, 8864
Telemetry [candidate] (8.82 ms) : 0, 8820
section iast
BytebuddyAgent [baseline] (838.262 ms) : 0, 838262
BytebuddyAgent [candidate] (829.462 ms) : 0, 829462
GlobalTracer [baseline] (311.388 ms) : 0, 311388
GlobalTracer [candidate] (309.976 ms) : 0, 309976
AppSec [baseline] (55.778 ms) : 0, 55778
AppSec [candidate] (56.551 ms) : 0, 56551
IAST [baseline] (20.909 ms) : 0, 20909
IAST [candidate] (20.983 ms) : 0, 20983
Remote Config [baseline] (592.6 µs) : 0, 593
Remote Config [candidate] (628.22 µs) : 0, 628
Telemetry [baseline] (8.275 ms) : 0, 8275
Telemetry [candidate] (8.292 ms) : 0, 8292
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (829.136 ms) : 0, 829136
BytebuddyAgent [candidate] (826.227 ms) : 0, 826227
GlobalTracer [baseline] (309.881 ms) : 0, 309881
GlobalTracer [candidate] (309.75 ms) : 0, 309750
AppSec [baseline] (57.379 ms) : 0, 57379
AppSec [candidate] (56.836 ms) : 0, 56836
IAST [baseline] (21.04 ms) : 0, 21040
IAST [candidate] (21.17 ms) : 0, 21170
Remote Config [baseline] (605.121 µs) : 0, 605
Remote Config [candidate] (609.801 µs) : 0, 610
Telemetry [baseline] (8.279 ms) : 0, 8279
Telemetry [candidate] (8.263 ms) : 0, 8263
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (824.429 ms) : 0, 824429
BytebuddyAgent [candidate] (823.452 ms) : 0, 823452
GlobalTracer [baseline] (309.669 ms) : 0, 309669
GlobalTracer [candidate] (308.234 ms) : 0, 308234
AppSec [baseline] (58.283 ms) : 0, 58283
AppSec [candidate] (57.751 ms) : 0, 57751
IAST [baseline] (20.826 ms) : 0, 20826
IAST [candidate] (20.695 ms) : 0, 20695
Remote Config [baseline] (613.568 µs) : 0, 614
Remote Config [candidate] (596.315 µs) : 0, 596
Telemetry [baseline] (8.229 ms) : 0, 8229
Telemetry [candidate] (8.149 ms) : 0, 8149
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.11 s) : 0, 1110341
Total [baseline] (10.456 s) : 0, 10455549
Agent [candidate] (1.122 s) : 0, 1122333
Total [candidate] (10.497 s) : 0, 10497382
section appsec
Agent [baseline] (1.247 s) : 0, 1247463
Total [baseline] (10.719 s) : 0, 10719019
Agent [candidate] (1.249 s) : 0, 1249354
Total [candidate] (10.731 s) : 0, 10730503
section iast
Agent [baseline] (1.238 s) : 0, 1238274
Total [baseline] (11.019 s) : 0, 11019439
Agent [candidate] (1.239 s) : 0, 1239402
Total [candidate] (10.998 s) : 0, 10997877
section profiling
Agent [baseline] (1.347 s) : 0, 1346635
Total [baseline] (10.943 s) : 0, 10942787
Agent [candidate] (1.34 s) : 0, 1339817
Total [candidate] (10.918 s) : 0, 10918273
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.11 s -
Agent appsec 1.247 s 137.122 ms (12.3%)
Agent iast 1.238 s 127.934 ms (11.5%)
Agent profiling 1.347 s 236.294 ms (21.3%)
Total tracing 10.456 s -
Total appsec 10.719 s 263.47 ms (2.5%)
Total iast 11.019 s 563.889 ms (5.4%)
Total profiling 10.943 s 487.238 ms (4.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.122 s -
Agent appsec 1.249 s 127.021 ms (11.3%)
Agent iast 1.239 s 117.069 ms (10.4%)
Agent profiling 1.34 s 217.484 ms (19.4%)
Total tracing 10.497 s -
Total appsec 10.731 s 233.122 ms (2.2%)
Total iast 10.998 s 500.496 ms (4.8%)
Total profiling 10.918 s 420.891 ms (4.0%) 
gantt
    title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.673 ms) : 0, 711673
BytebuddyAgent [candidate] (720.402 ms) : 0, 720402
GlobalTracer [baseline] (318.754 ms) : 0, 318754
GlobalTracer [candidate] (321.797 ms) : 0, 321797
AppSec [baseline] (55.386 ms) : 0, 55386
AppSec [candidate] (55.347 ms) : 0, 55347
Remote Config [baseline] (669.265 µs) : 0, 669
Remote Config [candidate] (678.797 µs) : 0, 679
Telemetry [baseline] (8.913 ms) : 0, 8913
Telemetry [candidate] (8.929 ms) : 0, 8929
section appsec
BytebuddyAgent [baseline] (728.806 ms) : 0, 728806
BytebuddyAgent [candidate] (730.401 ms) : 0, 730401
GlobalTracer [baseline] (315.77 ms) : 0, 315770
GlobalTracer [candidate] (315.88 ms) : 0, 315880
AppSec [baseline] (169.894 ms) : 0, 169894
AppSec [candidate] (168.563 ms) : 0, 168563
IAST [baseline] (19.12 ms) : 0, 19120
IAST [candidate] (20.664 ms) : 0, 20664
Remote Config [baseline] (654.379 µs) : 0, 654
Remote Config [candidate] (639.762 µs) : 0, 640
Telemetry [baseline] (7.902 ms) : 0, 7902
Telemetry [candidate] (8.6 ms) : 0, 8600
section iast
BytebuddyAgent [baseline] (827.376 ms) : 0, 827376
BytebuddyAgent [candidate] (828.293 ms) : 0, 828293
GlobalTracer [baseline] (307.638 ms) : 0, 307638
GlobalTracer [candidate] (308.719 ms) : 0, 308719
AppSec [baseline] (58.318 ms) : 0, 58318
AppSec [candidate] (57.411 ms) : 0, 57411
IAST [baseline] (21.142 ms) : 0, 21142
IAST [candidate] (21.124 ms) : 0, 21124
Remote Config [baseline] (603.583 µs) : 0, 604
Remote Config [candidate] (613.597 µs) : 0, 614
Telemetry [baseline] (8.283 ms) : 0, 8283
Telemetry [candidate] (8.297 ms) : 0, 8297
section profiling
BytebuddyAgent [baseline] (705.276 ms) : 0, 705276
BytebuddyAgent [candidate] (700.232 ms) : 0, 700232
GlobalTracer [baseline] (442.573 ms) : 0, 442573
GlobalTracer [candidate] (440.293 ms) : 0, 440293
AppSec [baseline] (53.998 ms) : 0, 53998
AppSec [candidate] (53.675 ms) : 0, 53675
Remote Config [baseline] (657.35 µs) : 0, 657
Remote Config [candidate] (674.224 µs) : 0, 674
Telemetry [baseline] (7.877 ms) : 0, 7877
Telemetry [candidate] (7.843 ms) : 0, 7843
ProfilingAgent [baseline] (94.341 ms) : 0, 94341
ProfilingAgent [candidate] (95.163 ms) : 0, 95163
Profiling [baseline] (94.365 ms) : 0, 94365
Profiling [candidate] (95.187 ms) : 0, 95187
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-12-20T08:39:28 2024-12-20T08:46:27
git_branch master smola/smoke-test-log-errors
git_commit_date 1734636365 1734683196
git_commit_sha 88c493c 66897ba
release_version 1.45.0-SNAPSHOT~88c493cc76 1.45.0-SNAPSHOT~66897bacec
start_time 2024-12-20T08:39:14 2024-12-20T08:46:14
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1734684741 1734684741
ci_job_id 746725582 746725582
ci_pipeline_id 51638544 51638544
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 15 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.366 ms) : 1347, 1386
.   : milestone, 1366,
appsec (1.741 ms) : 1718, 1765
.   : milestone, 1741,
appsec_no_iast (1.764 ms) : 1740, 1788
.   : milestone, 1764,
iast (1.499 ms) : 1476, 1522
.   : milestone, 1499,
profiling (1.517 ms) : 1492, 1541
.   : milestone, 1517,
tracing (1.493 ms) : 1469, 1517
.   : milestone, 1493,
section candidate
no_agent (1.351 ms) : 1331, 1371
.   : milestone, 1351,
appsec (1.755 ms) : 1732, 1778
.   : milestone, 1755,
appsec_no_iast (1.74 ms) : 1715, 1765
.   : milestone, 1740,
iast (1.495 ms) : 1472, 1519
.   : milestone, 1495,
profiling (1.545 ms) : 1520, 1571
.   : milestone, 1545,
tracing (1.49 ms) : 1466, 1514
.   : milestone, 1490,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.366 ms [1.347 ms, 1.386 ms] -
appsec 1.741 ms [1.718 ms, 1.765 ms] 374.634 µs (27.4%)
appsec_no_iast 1.764 ms [1.74 ms, 1.788 ms] 397.64 µs (29.1%)
iast 1.499 ms [1.476 ms, 1.522 ms] 132.72 µs (9.7%)
profiling 1.517 ms [1.492 ms, 1.541 ms] 150.052 µs (11.0%)
tracing 1.493 ms [1.469 ms, 1.517 ms] 126.567 µs (9.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.351 ms [1.331 ms, 1.371 ms] -
appsec 1.755 ms [1.732 ms, 1.778 ms] 403.896 µs (29.9%)
appsec_no_iast 1.74 ms [1.715 ms, 1.765 ms] 388.917 µs (28.8%)
iast 1.495 ms [1.472 ms, 1.519 ms] 144.191 µs (10.7%)
profiling 1.545 ms [1.52 ms, 1.571 ms] 194.449 µs (14.4%)
tracing 1.49 ms [1.466 ms, 1.514 ms] 139.389 µs (10.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76
    dateFormat X
    axisFormat %s
section baseline
no_agent (375.558 µs) : 355, 396
.   : milestone, 376,
iast (496.346 µs) : 475, 518
.   : milestone, 496,
iast_FULL (657.581 µs) : 636, 679
.   : milestone, 658,
iast_GLOBAL (529.433 µs) : 507, 552
.   : milestone, 529,
iast_HARDCODED_SECRET_DISABLED (496.35 µs) : 475, 518
.   : milestone, 496,
iast_INACTIVE (457.624 µs) : 436, 479
.   : milestone, 458,
iast_TELEMETRY_OFF (487.195 µs) : 465, 509
.   : milestone, 487,
tracing (452.755 µs) : 431, 474
.   : milestone, 453,
section candidate
no_agent (378.881 µs) : 359, 399
.   : milestone, 379,
iast (494.361 µs) : 472, 517
.   : milestone, 494,
iast_FULL (659.774 µs) : 638, 681
.   : milestone, 660,
iast_GLOBAL (517.272 µs) : 496, 539
.   : milestone, 517,
iast_HARDCODED_SECRET_DISABLED (491.211 µs) : 470, 512
.   : milestone, 491,
iast_INACTIVE (449.036 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (481.39 µs) : 460, 503
.   : milestone, 481,
tracing (446.764 µs) : 426, 467
.   : milestone, 447,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 375.558 µs [355.433 µs, 395.682 µs] -
iast 496.346 µs [474.899 µs, 517.792 µs] 120.788 µs (32.2%)
iast_FULL 657.581 µs [635.957 µs, 679.204 µs] 282.023 µs (75.1%)
iast_GLOBAL 529.433 µs [506.727 µs, 552.139 µs] 153.876 µs (41.0%)
iast_HARDCODED_SECRET_DISABLED 496.35 µs [474.568 µs, 518.133 µs] 120.793 µs (32.2%)
iast_INACTIVE 457.624 µs [435.93 µs, 479.319 µs] 82.067 µs (21.9%)
iast_TELEMETRY_OFF 487.195 µs [465.418 µs, 508.972 µs] 111.637 µs (29.7%)
tracing 452.755 µs [431.465 µs, 474.045 µs] 77.197 µs (20.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 378.881 µs [359.003 µs, 398.76 µs] -
iast 494.361 µs [472.01 µs, 516.711 µs] 115.479 µs (30.5%)
iast_FULL 659.774 µs [638.359 µs, 681.189 µs] 280.892 µs (74.1%)
iast_GLOBAL 517.272 µs [495.856 µs, 538.687 µs] 138.39 µs (36.5%)
iast_HARDCODED_SECRET_DISABLED 491.211 µs [469.95 µs, 512.471 µs] 112.329 µs (29.6%)
iast_INACTIVE 449.036 µs [428.498 µs, 469.575 µs] 70.155 µs (18.5%)
iast_TELEMETRY_OFF 481.39 µs [460.006 µs, 502.773 µs] 102.508 µs (27.1%)
tracing 446.764 µs [426.264 µs, 467.264 µs] 67.882 µs (17.9%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/smoke-test-log-errors
git_commit_date 1734636365 1734683196
git_commit_sha 88c493c 66897ba
release_version 1.45.0-SNAPSHOT~88c493cc76 1.45.0-SNAPSHOT~66897bacec
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1734685334 1734685334
ci_job_id 746725583 746725583
ci_pipeline_id 51638544 51638544
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.032 s) : 15032000, 15032000
.   : milestone, 15032000,
appsec (14.939 s) : 14939000, 14939000
.   : milestone, 14939000,
iast (18.59 s) : 18590000, 18590000
.   : milestone, 18590000,
iast_GLOBAL (18.075 s) : 18075000, 18075000
.   : milestone, 18075000,
profiling (15.233 s) : 15233000, 15233000
.   : milestone, 15233000,
tracing (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
section candidate
no_agent (14.951 s) : 14951000, 14951000
.   : milestone, 14951000,
appsec (14.943 s) : 14943000, 14943000
.   : milestone, 14943000,
iast (18.962 s) : 18962000, 18962000
.   : milestone, 18962000,
iast_GLOBAL (18.107 s) : 18107000, 18107000
.   : milestone, 18107000,
profiling (15.086 s) : 15086000, 15086000
.   : milestone, 15086000,
tracing (15.002 s) : 15002000, 15002000
.   : milestone, 15002000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.032 s [15.032 s, 15.032 s] -
appsec 14.939 s [14.939 s, 14.939 s] -93.0 ms (-0.6%)
iast 18.59 s [18.59 s, 18.59 s] 3.558 s (23.7%)
iast_GLOBAL 18.075 s [18.075 s, 18.075 s] 3.043 s (20.2%)
profiling 15.233 s [15.233 s, 15.233 s] 201.0 ms (1.3%)
tracing 15.106 s [15.106 s, 15.106 s] 74.0 ms (0.5%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.951 s [14.951 s, 14.951 s] -
appsec 14.943 s [14.943 s, 14.943 s] -8.0 ms (-0.1%)
iast 18.962 s [18.962 s, 18.962 s] 4.011 s (26.8%)
iast_GLOBAL 18.107 s [18.107 s, 18.107 s] 3.156 s (21.1%)
profiling 15.086 s [15.086 s, 15.086 s] 135.0 ms (0.9%)
tracing 15.002 s [15.002 s, 15.002 s] 51.0 ms (0.3%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~66897bacec, baseline=1.45.0-SNAPSHOT~88c493cc76
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.469 ms) : 1457, 1480
.   : milestone, 1469,
appsec (2.354 ms) : 2312, 2395
.   : milestone, 2354,
iast (2.089 ms) : 2037, 2142
.   : milestone, 2089,
iast_GLOBAL (2.136 ms) : 2083, 2189
.   : milestone, 2136,
profiling (1.968 ms) : 1925, 2011
.   : milestone, 1968,
tracing (1.94 ms) : 1899, 1980
.   : milestone, 1940,
section candidate
no_agent (1.47 ms) : 1458, 1481
.   : milestone, 1470,
appsec (2.353 ms) : 2312, 2395
.   : milestone, 2353,
iast (2.086 ms) : 2033, 2139
.   : milestone, 2086,
iast_GLOBAL (2.147 ms) : 2093, 2201
.   : milestone, 2147,
profiling (1.982 ms) : 1939, 2025
.   : milestone, 1982,
tracing (1.935 ms) : 1895, 1975
.   : milestone, 1935,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.469 ms [1.457 ms, 1.48 ms] -
appsec 2.354 ms [2.312 ms, 2.395 ms] 885.012 µs (60.3%)
iast 2.089 ms [2.037 ms, 2.142 ms] 620.773 µs (42.3%)
iast_GLOBAL 2.136 ms [2.083 ms, 2.189 ms] 667.622 µs (45.5%)
profiling 1.968 ms [1.925 ms, 2.011 ms] 499.132 µs (34.0%)
tracing 1.94 ms [1.899 ms, 1.98 ms] 471.122 µs (32.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.47 ms [1.458 ms, 1.481 ms] -
appsec 2.353 ms [2.312 ms, 2.395 ms] 883.952 µs (60.2%)
iast 2.086 ms [2.033 ms, 2.139 ms] 616.263 µs (41.9%)
iast_GLOBAL 2.147 ms [2.093 ms, 2.201 ms] 677.734 µs (46.1%)
profiling 1.982 ms [1.939 ms, 2.025 ms] 512.169 µs (34.9%)
tracing 1.935 ms [1.895 ms, 1.975 ms] 465.262 µs (31.7%)

@smola smola force-pushed the smola/smoke-test-log-errors branch 7 times, most recently from 112f4f1 to acfbbb9 Compare December 18, 2024 21:34
@smola smola marked this pull request as ready for review December 18, 2024 22:24
@smola smola requested review from a team as code owners December 18, 2024 22:24
PerfectSlayer
PerfectSlayer approved these changes Dec 19, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice improvements! 🙏

Comment thread dd-smoke-tests/src/main/groovy/datadog/smoketest/ProcessManager.groovy Outdated
Comment thread ...-tests/iast-util/src/testFixtures/groovy/datadog/smoketest/AbstractIastSpringBootTest.groovy Outdated
Comment thread dd-smoke-tests/java9-modules/src/test/groovy/datadog/smoketest/Java9ModulesSmokeTest.groovy Outdated
manuel-alvarez-alvarez
manuel-alvarez-alvarez reviewed Dec 19, 2024
View reviewed changes
Comment thread dd-smoke-tests/src/main/groovy/datadog/smoketest/ProcessManager.groovy Outdated
Copy link
Copy Markdown
Member

@manuel-alvarez-alvarez manuel-alvarez-alvarez Dec 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be very helpful if we could provide the logFilePaths in the assertions (although it's no minor change so don't bother 😄)

Copy link
Copy Markdown
Member Author

@smola smola Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not used that much, I went for something simple to use, but happy to change or adapt later.

manuel-alvarez-alvarez
manuel-alvarez-alvarez approved these changes Dec 19, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@manuel-alvarez-alvarez manuel-alvarez-alvarez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, to review the issue with the profiling test

@smola smola force-pushed the smola/smoke-test-log-errors branch 2 times, most recently from 2b1e6a3 to 9186742 Compare December 19, 2024 13:27
@smola smola requested a review from PerfectSlayer December 19, 2024 14:15
PerfectSlayer
PerfectSlayer approved these changes Dec 19, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for the changes 🙏

Comment thread dd-smoke-tests/src/main/groovy/datadog/smoketest/ProcessManager.groovy Outdated
@PerfectSlayer PerfectSlayer added the type: enhancement Enhancements and improvements label Dec 19, 2024
@smola smola force-pushed the smola/smoke-test-log-errors branch from 669c4b4 to 40b1393 Compare December 20, 2024 07:59
@smola smola merged commit ab205f6 into master Dec 20, 2024
@smola smola deleted the smola/smoke-test-log-errors branch December 20, 2024 09:48
@github-actions github-actions Bot added this to the 1.45.0 milestone Dec 20, 2024
@smola smola mentioned this pull request Dec 20, 2024
Merged
smola added a commit that referenced this pull request Dec 20, 2024
@smola
Revert "Check logs for errors at smoke tests cleanup (#8111)"
8c7b630 
This reverts commit ab205f6.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@manuel-alvarez-alvarez manuel-alvarez-alvarez manuel-alvarez-alvarez approved these changes

@Mariovido Mariovido Awaiting requested review from Mariovido Mariovido is a code owner automatically assigned from DataDog/asm-java

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes type: enhancement Enhancements and improvements

Projects

None yet

Milestone

1.45.0

Development

Successfully merging this pull request may close these issues.

3 participants

@smola @PerfectSlayer @manuel-alvarez-alvarez