Skip to content

Update GraphQL tests to take a memory dump when not shutting down#2655

Merged
andrewlock merged 2 commits into
masterfrom
andrew/capture-memory-dump-on-kill
Apr 11, 2022
Merged

Update GraphQL tests to take a memory dump when not shutting down#2655
andrewlock merged 2 commits into
masterfrom
andrew/capture-memory-dump-on-kill

Conversation

@andrewlock

@andrewlock andrewlock commented Apr 8, 2022

Copy link
Copy Markdown
Member

Summary of changes

  • If GraphQL process fails to exit, take a memory dump

Reason for change

We occasionally see the GraphQL process fail to exit correctly. This causes flaky test behaviour, because in that case we often don't receive the telemetry. Taking a memory dump will help us to diagnose the problem.

Implementation details

Downloads procdump and runs against the process. Crude, but it works, assuming the process isn't just a bit slow in shutting down. Now we play the waiting game.

Test coverage

Did a test in which I remove the shutdown endpoint in graphql, so it never shuts down. This generates the memory dump as expected, and uploads it inside the logs artifact for the job.

@andrewlock andrewlock added the area:tests unit tests, integration tests label Apr 8, 2022
@andrewlock
andrewlock requested a review from a team as a code owner April 8, 2022 12:23
@andrewlock

andrewlock commented Apr 8, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

✔️ Merging #2655 into master will not change line coverage
✔️ Merging #2655 into master will not change branch coverage
✔️ Merging #2655 into master will not change complexity

master #2655 Change
Lines 13438 / 18464 13453 / 18464
Lines % 73% 73% 0% ✔️
Branches 7754 / 11017 7765 / 11017
Branches % 70% 70% 0% ✔️
Complexity 12307 12307 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #2655 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 12307 12307 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change

View the full reports for further details:

@andrewlock

andrewlock commented Apr 8, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #2655 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.231
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 981μs 4.09μs 15.8μs 0 0 0 3.16 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 747μs 2.59μs 9.68μs 0 0 0 2.57 KB
#2655 WriteAndFlushEnrichedTraces net472 991μs 5.68μs 45.8μs 0 0 0 3.16 KB
#2655 WriteAndFlushEnrichedTraces netcoreapp3.1 782μs 4.52μs 35.3μs 0 0 0 2.57 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 320ns 1.84ns 13.5ns 0.0519 0 0 329 B
master AllCycleSimpleBody netcoreapp3.1 386ns 1.43ns 5.16ns 0.00446 0 0 328 B
master AllCycleMoreComplexBody net472 298ns 1.37ns 5.31ns 0.0482 0 0 305 B
master AllCycleMoreComplexBody netcoreapp3.1 376ns 1.38ns 4.97ns 0.00416 0 0 304 B
master BodyExtractorSimpleBody net472 451ns 2.35ns 12ns 0.0568 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 443ns 2.07ns 8.03ns 0.00377 0 0 272 B
master BodyExtractorMoreComplexBody net472 27.3μs 137ns 598ns 1.18 0.0127 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 21.7μs 96.1ns 372ns 0.0946 0 0 6.75 KB
#2655 AllCycleSimpleBody net472 303ns 0.867ns 3.13ns 0.052 0 0 329 B
#2655 AllCycleSimpleBody netcoreapp3.1 389ns 1.04ns 3.74ns 0.00456 0 0 328 B
#2655 AllCycleMoreComplexBody net472 305ns 0.727ns 2.72ns 0.0482 0 0 305 B
#2655 AllCycleMoreComplexBody netcoreapp3.1 375ns 1.35ns 5.22ns 0.00418 0 0 304 B
#2655 BodyExtractorSimpleBody net472 451ns 1.65ns 6.38ns 0.057 0 0 361 B
#2655 BodyExtractorSimpleBody netcoreapp3.1 458ns 2.34ns 10.7ns 0.00382 0 0 272 B
#2655 BodyExtractorMoreComplexBody net472 25.5μs 131ns 1.02μs 1.18 0.0127 0 7.62 KB
#2655 BodyExtractorMoreComplexBody netcoreapp3.1 21.9μs 106ns 423ns 0.086 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0.0139ns 0.0062ns 0.0416ns 0 0 0 0 b
master SendRequest netcoreapp3.1 341μs 814ns 3.15μs 0.168 0 0 19.74 KB
#2655 SendRequest net472 0.00059ns 0.00059ns 0.00307ns 0 0 0 0 b
#2655 SendRequest netcoreapp3.1 347μs 1.08μs 4.2μs 0.171 0 0 19.74 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.88μs 6.85ns 25.6ns 0.0941 0 0 594 B
master ExecuteNonQuery netcoreapp3.1 1.6μs 9.32ns 78ns 0.00845 0 0 632 B
#2655 ExecuteNonQuery net472 1.86μs 8.75ns 35ns 0.0939 0.000978 0 594 B
#2655 ExecuteNonQuery netcoreapp3.1 1.55μs 6.37ns 29.9ns 0.00935 0 0 632 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.82μs 9.7ns 37.6ns 0.127 0 0 803 B
master CallElasticsearch netcoreapp3.1 1.81μs 5.12ns 19.8ns 0.0109 0 0 792 B
master CallElasticsearchAsync net472 3.1μs 15ns 73.3ns 0.148 0.00154 0 939 B
master CallElasticsearchAsync netcoreapp3.1 1.91μs 5.43ns 20.3ns 0.0123 0 0 912 B
#2655 CallElasticsearch net472 2.99μs 14.5ns 59.9ns 0.127 0 0 803 B
#2655 CallElasticsearch netcoreapp3.1 1.81μs 10.3ns 73.9ns 0.0109 0 0 792 B
#2655 CallElasticsearchAsync net472 3.07μs 15ns 62.1ns 0.149 0 0 939 B
#2655 CallElasticsearchAsync netcoreapp3.1 1.96μs 3.78ns 13.6ns 0.0129 0 0 912 B
Benchmarks.Trace.GraphQLBenchmark - Faster 🎉 Same allocations ✔️

Faster 🎉 in #2655

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1 1.231 2,468.75 2,005.32

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 3.29μs 11.9ns 44.5ns 0.168 0.00163 0 1.06 KB
master ExecuteAsync netcoreapp3.1 2.47μs 9.34ns 33.7ns 0.0135 0 0 1.03 KB
#2655 ExecuteAsync net472 3.33μs 13.3ns 53.3ns 0.167 0.00171 0 1.06 KB
#2655 ExecuteAsync netcoreapp3.1 2μs 4.75ns 16.4ns 0.0146 0 0 1.03 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 7.47μs 42.7ns 331ns 0.354 0 0 2.28 KB
master SendAsync netcoreapp3.1 5.1μs 25.2ns 104ns 0.0303 0 0 2.21 KB
#2655 SendAsync net472 7.26μs 33.8ns 135ns 0.361 0 0 2.28 KB
#2655 SendAsync netcoreapp3.1 5.66μs 32.8ns 279ns 0.0296 0 0 2.21 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.54μs 18.1ns 78.7ns 0.228 0 0 1.45 KB
master EnrichedLog netcoreapp3.1 3.41μs 18.6ns 102ns 0.0204 0 0 1.53 KB
#2655 EnrichedLog net472 3.74μs 21.5ns 184ns 0.227 0 0 1.45 KB
#2655 EnrichedLog netcoreapp3.1 3.19μs 7.67ns 28.7ns 0.0208 0 0 1.53 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 303μs 640ns 2.39μs 0.456 0.152 0 4.33 KB
master EnrichedLog netcoreapp3.1 247μs 1.42μs 10.6μs 0 0 0 4.21 KB
#2655 EnrichedLog net472 305μs 1.19μs 4.43μs 0.453 0.151 0 4.33 KB
#2655 EnrichedLog netcoreapp3.1 247μs 1.21μs 8.7μs 0 0 0 4.21 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 8.46μs 29.6ns 111ns 0.506 0 0 3.23 KB
master EnrichedLog netcoreapp3.1 6.6μs 32.1ns 128ns 0.0502 0 0 3.6 KB
#2655 EnrichedLog net472 8.15μs 23.9ns 92.7ns 0.503 0 0 3.23 KB
#2655 EnrichedLog netcoreapp3.1 6.76μs 31ns 112ns 0.0495 0 0 3.6 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.52μs 12.6ns 58ns 0.161 0.00123 0 1.01 KB
master SendReceive netcoreapp3.1 2.11μs 8.37ns 32.4ns 0.0137 0 0 1.01 KB
#2655 SendReceive net472 2.43μs 11.4ns 46.9ns 0.16 0.00117 0 1.01 KB
#2655 SendReceive netcoreapp3.1 2.24μs 12.4ns 79.5ns 0.0136 0 0 1.01 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 6.28μs 25.3ns 98.2ns 0.292 0 0 1.87 KB
master EnrichedLog netcoreapp3.1 5.39μs 17ns 85ns 0.0192 0 0 1.49 KB
#2655 EnrichedLog net472 6.72μs 43.3ns 430ns 0.29 0 0 1.87 KB
#2655 EnrichedLog netcoreapp3.1 5.55μs 27ns 121ns 0.0195 0 0 1.49 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 931ns 2.92ns 10.9ns 0.0718 0 0 457 B
master StartFinishSpan netcoreapp3.1 955ns 5.53ns 45.6ns 0.00626 0 0 456 B
master StartFinishScope net472 1.15μs 5.06ns 18.9ns 0.0843 0 0 538 B
master StartFinishScope netcoreapp3.1 1.11μs 5.94ns 43.2ns 0.00802 0 0 576 B
#2655 StartFinishSpan net472 940ns 2.89ns 11.2ns 0.0719 0 0 457 B
#2655 StartFinishSpan netcoreapp3.1 923ns 3.74ns 14.5ns 0.00638 0 0 456 B
#2655 StartFinishScope net472 1.17μs 4.45ns 17.2ns 0.0842 0 0 538 B
#2655 StartFinishScope netcoreapp3.1 1.13μs 6.01ns 43.3ns 0.00778 0 0 576 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.34μs 6.1ns 23.6ns 0.084 0 0 538 B
master RunOnMethodBegin netcoreapp3.1 1.24μs 5.82ns 22.5ns 0.00747 0 0 576 B
#2655 RunOnMethodBegin net472 1.37μs 6.6ns 27.2ns 0.0839 0 0 538 B
#2655 RunOnMethodBegin netcoreapp3.1 1.21μs 5.3ns 19.8ns 0.00781 0 0 576 B

@andrewlock
andrewlock merged commit 439af9c into master Apr 11, 2022
@andrewlock
andrewlock deleted the andrew/capture-memory-dump-on-kill branch April 11, 2022 20:20
@github-actions github-actions Bot added this to the vNext milestone Apr 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:tests unit tests, integration tests

Projects

None yet

Milestone

2.7.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @zacharycmontoya @pierotibou