Skip to content

[ASM] Increase waf timeout for all integration tests#3124

Merged
anna-git merged 1 commit into
masterfrom
asm/anna/generalize-waf-timeout-tests
Aug 25, 2022
Merged

[ASM] Increase waf timeout for all integration tests#3124
anna-git merged 1 commit into
masterfrom
asm/anna/generalize-waf-timeout-tests

Conversation

@anna-git

@anna-git anna-git commented Aug 25, 2022

Copy link
Copy Markdown
Contributor

Summary of changes

The custom waf timeout of 1 second was only set for aspnet core integration tests.
Increase it to 10 seconds and for all tests, including IIS express ones.

Fix security solution filter, for small changes like that, we would need it

Reason for change

Timed out on CI pipeline

Implementation details

Test coverage

Other details

@anna-git
anna-git requested review from a team as code owners August 25, 2022 09:32
andrewlock
andrewlock approved these changes Aug 25, 2022
View reviewed changes

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@andrewlock

andrewlock commented Aug 25, 2022

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3124 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 718μs 208ns 748ns 0.361 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 461μs 123ns 460ns 0 0 0 2.59 KB
#3124 WriteAndFlushEnrichedTraces net472 714μs 386ns 1.49μs 0.355 0 0 3.18 KB
#3124 WriteAndFlushEnrichedTraces netcoreapp3.1 455μs 185ns 639ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 186ns 0.201ns 0.78ns 0.0676 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 238ns 0.403ns 1.56ns 0.00583 0 0 424 B
master AllCycleMoreComplexBody net472 188ns 0.158ns 0.61ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 236ns 0.243ns 0.941ns 0.00538 0 0 400 B
master BodyExtractorSimpleBody net472 263ns 0.237ns 0.888ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 224ns 0.289ns 1.04ns 0.00381 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.6μs 9.98ns 38.7ns 1.21 0.0219 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.7μs 11.6ns 43.2ns 0.0891 0 0 6.75 KB
#3124 AllCycleSimpleBody net472 186ns 0.236ns 0.881ns 0.0675 0 0 425 B
#3124 AllCycleSimpleBody netcoreapp3.1 238ns 0.346ns 1.34ns 0.00576 0 0 424 B
#3124 AllCycleMoreComplexBody net472 183ns 0.164ns 0.614ns 0.0637 0 0 401 B
#3124 AllCycleMoreComplexBody netcoreapp3.1 236ns 0.273ns 1.02ns 0.00541 0 0 400 B
#3124 BodyExtractorSimpleBody net472 267ns 0.295ns 1.14ns 0.0573 0 0 361 B
#3124 BodyExtractorSimpleBody netcoreapp3.1 232ns 0.113ns 0.424ns 0.00373 0 0 272 B
#3124 BodyExtractorMoreComplexBody net472 15.5μs 10.2ns 39.6ns 1.2 0.0153 0 7.62 KB
#3124 BodyExtractorMoreComplexBody netcoreapp3.1 12.2μs 16.4ns 61.4ns 0.0914 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 181μs 142ns 548ns 0.27 0 0 20.57 KB
#3124 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3124 SendRequest netcoreapp3.1 180μs 208ns 805ns 0.269 0 0 20.57 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.78μs 0.664ns 2.49ns 0.15 0.000892 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.44μs 0.491ns 1.77ns 0.0124 0 0 936 B
#3124 ExecuteNonQuery net472 1.85μs 0.335ns 1.3ns 0.15 0.000916 0 947 B
#3124 ExecuteNonQuery netcoreapp3.1 1.44μs 0.508ns 1.9ns 0.0122 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.57μs 0.393ns 1.47ns 0.182 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.48μs 0.807ns 3.13ns 0.0148 0 0 1.1 KB
master CallElasticsearchAsync net472 2.57μs 0.494ns 1.85ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.65μs 0.575ns 2.23ns 0.0158 0 0 1.22 KB
#3124 CallElasticsearch net472 2.51μs 0.358ns 1.24ns 0.183 0 0 1.16 KB
#3124 CallElasticsearch netcoreapp3.1 1.55μs 1.09ns 4.21ns 0.0147 0 0 1.1 KB
#3124 CallElasticsearchAsync net472 2.65μs 0.675ns 2.53ns 0.205 0 0 1.29 KB
#3124 CallElasticsearchAsync netcoreapp3.1 1.56μs 0.45ns 1.68ns 0.0166 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.69μs 8.21ns 31.8ns 0.224 0 0 1.41 KB
master ExecuteAsync netcoreapp3.1 1.68μs 1.65ns 6.16ns 0.0184 0 0 1.34 KB
#3124 ExecuteAsync net472 2.62μs 4.11ns 15.9ns 0.224 0 0 1.41 KB
#3124 ExecuteAsync netcoreapp3.1 1.65μs 3.96ns 15.3ns 0.0184 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.88μs 7.79ns 30.2ns 0.437 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.57μs 6.35ns 24.6ns 0.0353 0 0 2.6 KB
#3124 SendAsync net472 5.55μs 17.2ns 64.3ns 0.439 0 0 2.77 KB
#3124 SendAsync netcoreapp3.1 3.5μs 5.31ns 19.9ns 0.0347 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.11μs 4.78ns 17.9ns 0.288 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.44μs 1.86ns 7.22ns 0.0256 0 0 1.85 KB
#3124 EnrichedLog net472 3.2μs 0.793ns 3.07ns 0.288 0 0 1.81 KB
#3124 EnrichedLog netcoreapp3.1 2.44μs 1.5ns 5.63ns 0.0258 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 150μs 192ns 745ns 0.674 0.225 0 4.65 KB
master EnrichedLog netcoreapp3.1 115μs 102ns 381ns 0.0577 0 0 4.49 KB
#3124 EnrichedLog net472 152μs 111ns 431ns 0.674 0.225 0 4.65 KB
#3124 EnrichedLog netcoreapp3.1 116μs 252ns 974ns 0.0578 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.54μs 19.8ns 76.5ns 0.568 0.00281 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.4μs 8.99ns 34.8ns 0.0531 0 0 3.91 KB
#3124 EnrichedLog net472 5.63μs 7.98ns 29.9ns 0.569 0.00279 0 3.59 KB
#3124 EnrichedLog netcoreapp3.1 4.37μs 12.4ns 48.1ns 0.0539 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.25μs 1.3ns 4.87ns 0.218 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.78μs 0.897ns 3.47ns 0.0178 0 0 1.32 KB
#3124 SendReceive net472 2.26μs 0.365ns 1.41ns 0.217 0 0 1.37 KB
#3124 SendReceive netcoreapp3.1 1.75μs 0.556ns 2.15ns 0.0181 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.92μs 1.33ns 4.99ns 0.352 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.27μs 1.7ns 6.57ns 0.0234 0 0 1.8 KB
#3124 EnrichedLog net472 4.95μs 2.03ns 7.84ns 0.352 0 0 2.23 KB
#3124 EnrichedLog netcoreapp3.1 4.29μs 2ns 7.74ns 0.0236 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.11μs 0.427ns 1.6ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 910ns 0.193ns 0.695ns 0.0103 0 0 760 B
master StartFinishScope net472 1.42μs 0.53ns 1.98ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.13μs 0.665ns 2.49ns 0.0118 0 0 880 B
#3124 StartFinishSpan net472 1.08μs 0.206ns 0.77ns 0.128 0 0 810 B
#3124 StartFinishSpan netcoreapp3.1 909ns 0.545ns 1.96ns 0.0104 0 0 760 B
#3124 StartFinishScope net472 1.37μs 0.302ns 1.17ns 0.141 0 0 891 B
#3124 StartFinishScope netcoreapp3.1 1.07μs 0.461ns 1.79ns 0.0114 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.53μs 0.826ns 2.98ns 0.142 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.17μs 1.29ns 5ns 0.012 0 0 880 B
#3124 RunOnMethodBegin net472 1.42μs 0.502ns 1.94ns 0.141 0 0 891 B
#3124 RunOnMethodBegin netcoreapp3.1 1.12μs 0.41ns 1.53ns 0.0121 0 0 880 B

@andrewlock

andrewlock commented Aug 25, 2022

Copy link
Copy Markdown
Member

Code Coverage Report 📊

✔️ Merging #3124 into master will not change line coverage
✔️ Merging #3124 into master will not change branch coverage
✔️ Merging #3124 into master will not change complexity

master #3124 Change
Lines 17190 / 23534 17173 / 23534
Lines % 73% 73% 0% ✔️
Branches 10221 / 14538 10214 / 14538
Branches % 70% 70% 0% ✔️
Complexity 15640 15640 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3124 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 15640 15640 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.CIVisibility 5% ✔️ 6% ✔️ 0 ✔️

View the full reports for further details:

dromanol
dromanol approved these changes Aug 25, 2022
View reviewed changes

@dromanol dromanol left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great :)

@anna-git
anna-git merged commit 0d3b262 into master Aug 25, 2022
@anna-git
anna-git deleted the asm/anna/generalize-waf-timeout-tests branch August 25, 2022 11:15
@github-actions github-actions Bot added this to the vNext milestone Aug 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewlock andrewlock andrewlock approved these changes
@dromanol dromanol dromanol approved these changes

Assignees

No one assigned

Labels

area:asm

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@anna-git @andrewlock @dromanol