Skip to content

[Snapshots] Print a small diff at the end of the tests#3098

Merged
pierotibou merged 5 commits into
masterfrom
pierre/diff-snaps-from-tests
Sep 19, 2022
Merged

[Snapshots] Print a small diff at the end of the tests#3098
pierotibou merged 5 commits into
masterfrom
pierre/diff-snaps-from-tests

Conversation

@pierotibou

@pierotibou pierotibou commented Aug 18, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary of changes

Just print a smalll diff at the end of integration tests. This one should be more useful to the developer, whereas the one in the PR, more useful for the reviewer.
While I was at it, I've added an extra commit to parallellize getting the snapshots from the build.

Reason for change

To have a short summary of diffs directly in the test output. The current received vs generated output after the test wasn't super helpful

Implementation details

Just use a diff library, run a target after the integration tests even if they fail. Everything happends behind the scene with Nuke.
Also, in the target UpdateSnapshotsFromBuild started download/unzip tasks in parallel.

Test coverage

Tested here

@pierotibou
pierotibou requested review from a team as code owners August 18, 2022 15:59
@github-actions github-actions Bot added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Aug 18, 2022
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

andrewlock commented Aug 21, 2022

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3098 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 719μs 1.13μs 4.39μs 0.359 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 458μs 563ns 2.18μs 0 0 0 2.59 KB
#3098 WriteAndFlushEnrichedTraces net472 727μs 496ns 1.85μs 0.365 0 0 3.18 KB
#3098 WriteAndFlushEnrichedTraces netcoreapp3.1 476μs 237ns 919ns 0 0 0 2.59 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 182ns 0.128ns 0.479ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 238ns 0.265ns 1.03ns 0.00587 0 0 424 B
master AllCycleMoreComplexBody net472 202ns 0.273ns 1.06ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 235ns 0.298ns 1.12ns 0.00542 0 0 400 B
master BodyExtractorSimpleBody net472 277ns 0.514ns 1.99ns 0.0574 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 227ns 0.353ns 1.32ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.8μs 70.9ns 292ns 1.21 0.0219 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.8μs 13ns 46.7ns 0.0883 0 0 6.75 KB
#3098 AllCycleSimpleBody net472 183ns 0.228ns 0.882ns 0.0676 0 0 425 B
#3098 AllCycleSimpleBody netcoreapp3.1 238ns 0.255ns 0.954ns 0.00581 0 0 424 B
#3098 AllCycleMoreComplexBody net472 182ns 0.134ns 0.5ns 0.0637 0 0 401 B
#3098 AllCycleMoreComplexBody netcoreapp3.1 236ns 0.285ns 1.11ns 0.00545 0 0 400 B
#3098 BodyExtractorSimpleBody net472 259ns 0.219ns 0.849ns 0.0574 0 0 361 B
#3098 BodyExtractorSimpleBody netcoreapp3.1 225ns 0.317ns 1.19ns 0.00362 0 0 272 B
#3098 BodyExtractorMoreComplexBody net472 14.3μs 5.12ns 19.2ns 1.2 0.0214 0 7.62 KB
#3098 BodyExtractorMoreComplexBody netcoreapp3.1 12.4μs 18.5ns 71.8ns 0.0929 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0.00189ns 0.00143ns 0.00536ns 0 0 0 0 b
master SendRequest netcoreapp3.1 178μs 365ns 1.42μs 0.178 0 0 20.33 KB
#3098 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3098 SendRequest netcoreapp3.1 178μs 98.1ns 367ns 0.177 0 0 20.33 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.64μs 3.06ns 11.8ns 0.126 0.000799 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.25μs 1.32ns 5.1ns 0.0114 0 0 824 B
#3098 ExecuteNonQuery net472 1.56μs 1.06ns 3.98ns 0.126 0.00078 0 794 B
#3098 ExecuteNonQuery netcoreapp3.1 1.27μs 0.58ns 2.17ns 0.0111 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.22μs 0.464ns 1.8ns 0.159 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.39μs 0.661ns 2.38ns 0.0127 0 0 984 B
master CallElasticsearchAsync net472 2.25μs 3.65ns 14.1ns 0.18 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.5μs 0.521ns 1.95ns 0.0151 0 0 1.1 KB
#3098 CallElasticsearch net472 2.19μs 0.457ns 1.77ns 0.159 0 0 1 KB
#3098 CallElasticsearch netcoreapp3.1 1.41μs 1.06ns 3.98ns 0.0134 0 0 984 B
#3098 CallElasticsearchAsync net472 2.39μs 0.589ns 2.2ns 0.18 0 0 1.14 KB
#3098 CallElasticsearchAsync netcoreapp3.1 1.45μs 0.526ns 1.97ns 0.0152 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.41μs 5.59ns 21.6ns 0.199 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.55μs 2.26ns 8.77ns 0.0161 0 0 1.22 KB
#3098 ExecuteAsync net472 2.39μs 6.46ns 25ns 0.2 0 0 1.26 KB
#3098 ExecuteAsync netcoreapp3.1 1.55μs 3.62ns 13.5ns 0.0165 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 4.98μs 8.67ns 33.6ns 0.392 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.32μs 4.37ns 16.9ns 0.0313 0 0 2.36 KB
#3098 SendAsync net472 4.86μs 8.88ns 34.4ns 0.393 0 0 2.48 KB
#3098 SendAsync netcoreapp3.1 3.17μs 3.23ns 11.2ns 0.0316 0 0 2.36 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.83μs 0.728ns 2.72ns 0.263 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.32μs 0.352ns 1.32ns 0.023 0 0 1.73 KB
#3098 EnrichedLog net472 2.81μs 1.02ns 3.94ns 0.263 0 0 1.66 KB
#3098 EnrichedLog netcoreapp3.1 2.32μs 1.59ns 5.96ns 0.0232 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 149μs 129ns 498ns 0.675 0.225 0 4.5 KB
master EnrichedLog netcoreapp3.1 117μs 124ns 481ns 0 0 0 4.38 KB
#3098 EnrichedLog net472 149μs 198ns 766ns 0.677 0.226 0 4.5 KB
#3098 EnrichedLog netcoreapp3.1 114μs 136ns 526ns 0.0571 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.38μs 7.92ns 27.4ns 0.545 0.0027 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.14μs 7.15ns 26.8ns 0.0526 0 0 3.8 KB
#3098 EnrichedLog net472 5.36μs 8.8ns 34.1ns 0.544 0.00265 0 3.43 KB
#3098 EnrichedLog netcoreapp3.1 4.16μs 10.6ns 41ns 0.052 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 1.98μs 0.628ns 2.43ns 0.193 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.67μs 0.668ns 2.5ns 0.0159 0 0 1.21 KB
#3098 SendReceive net472 1.97μs 0.559ns 2.09ns 0.193 0 0 1.22 KB
#3098 SendReceive netcoreapp3.1 1.66μs 0.52ns 2.01ns 0.0158 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.86μs 1.47ns 5.48ns 0.328 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4μs 2.06ns 7.71ns 0.0219 0 0 1.69 KB
#3098 EnrichedLog net472 4.69μs 0.998ns 3.73ns 0.329 0 0 2.08 KB
#3098 EnrichedLog netcoreapp3.1 4.21μs 2.53ns 9.48ns 0.0214 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 841ns 0.866ns 3.36ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 736ns 0.77ns 2.98ns 0.00886 0 0 648 B
master StartFinishScope net472 1.1μs 1.26ns 4.87ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 846ns 0.68ns 2.63ns 0.0102 0 0 768 B
#3098 StartFinishSpan net472 857ns 1.7ns 6.38ns 0.104 0 0 658 B
#3098 StartFinishSpan netcoreapp3.1 737ns 0.693ns 2.59ns 0.00884 0 0 648 B
#3098 StartFinishScope net472 1.07μs 1.42ns 5.32ns 0.117 0 0 738 B
#3098 StartFinishScope netcoreapp3.1 871ns 0.648ns 2.51ns 0.0104 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.27μs 1.21ns 4.69ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 953ns 1.05ns 3.92ns 0.0105 0 0 768 B
#3098 RunOnMethodBegin net472 1.19μs 1.85ns 7.15ns 0.117 0 0 738 B
#3098 RunOnMethodBegin netcoreapp3.1 1.02μs 0.673ns 2.52ns 0.01 0 0 768 B

@andrewlock

andrewlock commented Aug 21, 2022

Copy link
Copy Markdown
Member

Code Coverage Report 📊

⚠️ Merging #3098 into master will will decrease line coverage by 1%
⚠️ Merging #3098 into master will will decrease branch coverage by 2%
✔️ Merging #3098 into master will not change complexity

master #3098 Change
Lines 16872 / 23280 16607 / 23280
Lines % 72% 71% -1% ⚠️
Branches 10067 / 14362 9827 / 14362
Branches % 70% 68% -2% ⚠️
Complexity 15458 15458 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ⚠️

master #3098 Change
Lines % 72% 71% -1% ⚠️
Branches % 70% 68% -2% ⚠️
Complexity 15458 15458 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.ClrProfiler.AutoInstrumentation.Msmq.MessageQueue_ReceiveCurrent_Integration -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Msmq.MessageQueue_SendInternal_Integration -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Wcf.ChannelHandlerIntegration -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Http.HttpClient.WinHttpHandler.WinHttpHandlerIntegration -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Msmq.MessageQueue_Purge_Integration -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Wcf.SyncMethodInvokerIntegration -88% ⛔ -80% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Wcf.TaskMethodInvokerIntegration -88% ⛔ -80% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Wcf.WcfCommon -84% ⛔ -60% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Msmq.MsmqCommon -83% ⛔ -75% ⛔ 0 ✔️
Datadog.Trace.ClrProfiler.AutoInstrumentation.Logging.NLog.LogsInjection.DiagnosticContextHelper -56% ⛔ -48% ⛔ 0 ✔️
...And 23 more

View the full reports for further details:

@pierotibou
pierotibou merged commit 9643267 into master Sep 19, 2022
@pierotibou
pierotibou deleted the pierre/diff-snaps-from-tests branch September 19, 2022 15:42
@github-actions github-actions Bot added this to the vNext milestone Sep 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucaspimentel lucaspimentel lucaspimentel approved these changes
@anna-git anna-git anna-git approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

4 participants

@pierotibou @andrewlock @lucaspimentel @anna-git