Skip to content

[tests] show only diff in test failures when comparing long multi-line strings#3205

Merged
lucaspimentel merged 3 commits into
masterfrom
lpimentel/string-diffs-in-tests
Sep 15, 2022
Merged

[tests] show only diff in test failures when comparing long multi-line strings#3205
lucaspimentel merged 3 commits into
masterfrom
lpimentel/string-diffs-in-tests

Conversation

@lucaspimentel

@lucaspimentel lucaspimentel commented Sep 12, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Summary of changes

Add an extension for Fluent Assertions that compares strings for equality but only outputs a diff between the strings. Useful when working with very long multi-line strings.

For example, adding one assembly references causes the test AssemblyReferencesHaveNotChanged to fail with the following message:

Expected referencedAssemblyOutput to be "Microsoft.AspNetCore.Http.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Http.Extensions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Http.Features, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Mvc.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Routing, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Routing.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
[...57 lines total...]
" with a length of 5452 because Assembly references should match the verified list of assembly references. Update the verified snapshot when the assembly references change, but "Microsoft.AspNetCore.Http.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Http.Extensions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Http.Features, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Mvc.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Routing, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
Microsoft.AspNetCore.Routing.Abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60
[...58 lines total...]
" has a length of 5545, differs near "ool" (index 1896).

The message above includes the full text of both strings, but does not indicate what the differences are between the strings. After this PR, the message shows only the differences between the two strings, including their line number. In the example below it is clear that a single line was added at line 20 and the new assembly is System.Diagnostics.Tools:

ReferencedAssemblyOutput has differences from expected value because Assembly references should match the verified list of assembly references. Update the verified snapshot when the assembly references change. Diff:
+20:System.Diagnostics.Tools, Version=4.1.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a

Reason for change

This change should make it easier to diagnose failing test that compare long multi-line strings.

Implementation details

  • added DiffPlex library to perform the actual diff between strings
  • added StringAssertions.Be() extension method overload with additional bool outputDiffOnly parameter
  • added DiffPaneModelFormatter : IValueFormatter to convert the DiffPaneModel to output string used by Fluent Assertions
  • use new extension method to compare strings in PublicApiHasNotChanged and AssemblyReferencesHaveNotChanged

Usage:

// to output only the diff between both strings
actualString.Should().Be(expectedString, outputDiffOnly: true);

// to output both strings fully
actualString.Should().Be(expectedString, outputDiffOnly: false);

// internally, `showDiffOnly: false` simply calls the original `Be()`
actualString.Should().Be(expectedString);

Currently this only performs a line-by-line diff, but it would be trivial to add a parameter to the extension method to use different DiffPlex "chunkers" (e.g. word-by-word, char-by-char).

Test coverage

:inception:

@github-actions github-actions Bot added the area:tests unit tests, integration tests label Sep 12, 2022
@andrewlock

This comment has been minimized.

Sign in to view
@lucaspimentel lucaspimentel changed the title [tests] show only diff when comparing very long strings [tests] show only diff in test failures when comparing long multi-line strings Sep 12, 2022
@lucaspimentel
lucaspimentel marked this pull request as ready for review September 12, 2022 20:43
@lucaspimentel
lucaspimentel requested a review from a team as a code owner September 12, 2022 20:43
@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock

This comment has been minimized.

Sign in to view
andrewlock
andrewlock approved these changes Sep 13, 2022
View reviewed changes

@andrewlock andrewlock left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 👍

Comment thread tracer/test/Datadog.Trace.TestHelpers/FluentAssertionsExtensions/StringExtensions.cs Outdated
@lucaspimentel lucaspimentel added the type:enhancement Improvement to an existing feature label Sep 13, 2022
@andrewlock

This comment has been minimized.

Sign in to view
@lucaspimentel
lucaspimentel force-pushed the lpimentel/string-diffs-in-tests branch from 9f1f978 to e8af5d6 Compare September 14, 2022 17:02
@andrewlock

andrewlock commented Sep 15, 2022

Copy link
Copy Markdown
Member

Code Coverage Report 📊

✔️ Merging #3205 into master will not change line coverage
✔️ Merging #3205 into master will not change branch coverage
✔️ Merging #3205 into master will will decrease complexity by 2

master #3205 Change
Lines 18036 / 24569 18007 / 24566
Lines % 73% 73% 0% ✔️
Branches 10614 / 15058 10597 / 15056
Branches % 70% 70% 0% ✔️
Complexity 16282 16280 -2 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3205 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 16282 16280 -2 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Activity.DiagnosticSourceEventListener -6% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Ci.CIVisibility 5% ✔️ 6% ✔️ 0 ✔️

View the full reports for further details:

@andrewlock

andrewlock commented Sep 15, 2022

Copy link
Copy Markdown
Member

Benchmarks Report 🐌

Benchmarks for #3205 compared to master:

  • 2 benchmarks are slower, with geometric mean 1.214
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 715μs 1.26μs 4.9μs 0.359 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 458μs 620ns 2.4μs 0 0 0 2.58 KB
#3205 WriteAndFlushEnrichedTraces net472 782μs 4.22μs 23.1μs 0.374 0 0 3.18 KB
#3205 WriteAndFlushEnrichedTraces netcoreapp3.1 496μs 3.36μs 33.5μs 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 185ns 0.152ns 0.588ns 0.0675 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 238ns 0.242ns 0.904ns 0.00583 0 0 424 B
master AllCycleMoreComplexBody net472 180ns 0.105ns 0.379ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 235ns 0.446ns 1.61ns 0.00553 0 0 400 B
master BodyExtractorSimpleBody net472 261ns 0.301ns 1.13ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 215ns 0.197ns 0.737ns 0.0038 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.8μs 14.7ns 57.1ns 1.21 0.0147 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.1μs 9.75ns 37.7ns 0.0905 0 0 6.75 KB
#3205 AllCycleSimpleBody net472 183ns 0.141ns 0.528ns 0.0675 0 0 425 B
#3205 AllCycleSimpleBody netcoreapp3.1 235ns 0.225ns 0.872ns 0.00577 0 0 424 B
#3205 AllCycleMoreComplexBody net472 182ns 0.165ns 0.617ns 0.0637 0 0 401 B
#3205 AllCycleMoreComplexBody netcoreapp3.1 232ns 0.26ns 1.01ns 0.00549 0 0 400 B
#3205 BodyExtractorSimpleBody net472 263ns 0.285ns 1.07ns 0.0574 0 0 361 B
#3205 BodyExtractorSimpleBody netcoreapp3.1 211ns 0.309ns 1.16ns 0.0037 0 0 272 B
#3205 BodyExtractorMoreComplexBody net472 14.6μs 9.2ns 35.6ns 1.21 0.0145 0 7.62 KB
#3205 BodyExtractorMoreComplexBody netcoreapp3.1 12.1μs 16.9ns 63.1ns 0.0904 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0.000716ns 0.000625ns 0.00242ns 0 0 0 0 b
master SendRequest netcoreapp3.1 179μs 396ns 1.53μs 0.178 0 0 20.59 KB
#3205 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3205 SendRequest netcoreapp3.1 178μs 223ns 836ns 0.265 0 0 20.61 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.84μs 3.9ns 15.1ns 0.15 0 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.44μs 2.05ns 7.66ns 0.0122 0 0 936 B
#3205 ExecuteNonQuery net472 1.79μs 1.72ns 6.66ns 0.15 0.000901 0 947 B
#3205 ExecuteNonQuery netcoreapp3.1 1.41μs 0.61ns 2.28ns 0.0126 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #3205

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑netcoreapp3.1 1.187 1,565.35 1,858.41 several?

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.57μs 1.41ns 5.47ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.57μs 0.685ns 2.65ns 0.0149 0 0 1.1 KB
master CallElasticsearchAsync net472 2.77μs 1.32ns 5.11ns 0.204 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.61μs 0.577ns 2.16ns 0.0162 0 0 1.22 KB
#3205 CallElasticsearch net472 2.9μs 16.1ns 102ns 0.182 0 0 1.16 KB
#3205 CallElasticsearch netcoreapp3.1 1.85μs 18.5ns 184ns 0.0146 0.000861 0 1.1 KB
#3205 CallElasticsearchAsync net472 2.95μs 16.5ns 103ns 0.205 0 0 1.29 KB
#3205 CallElasticsearchAsync netcoreapp3.1 1.62μs 1.05ns 4.06ns 0.0164 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #3205

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1 1.240 1,702.70 2,112.13

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.82μs 8.46ns 32.8ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.7μs 0.893ns 3.34ns 0.0186 0 0 1.34 KB
#3205 ExecuteAsync net472 3.16μs 18.4ns 160ns 0.225 0 0 1.42 KB
#3205 ExecuteAsync netcoreapp3.1 2.12μs 14.9ns 146ns 0.0179 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.88μs 2.65ns 10.3ns 0.438 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.73μs 0.904ns 3.38ns 0.0355 0 0 2.6 KB
#3205 SendAsync net472 5.62μs 6.63ns 25.7ns 0.438 0 0 2.77 KB
#3205 SendAsync netcoreapp3.1 3.55μs 4.63ns 17.3ns 0.0359 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.16μs 3.56ns 13.3ns 0.287 0.00157 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.5μs 1.3ns 4.68ns 0.025 0 0 1.85 KB
#3205 EnrichedLog net472 3.17μs 3.07ns 11.5ns 0.288 0 0 1.81 KB
#3205 EnrichedLog netcoreapp3.1 2.51μs 1.47ns 5.5ns 0.0253 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 152μs 136ns 528ns 0.679 0.226 0 4.66 KB
master EnrichedLog netcoreapp3.1 119μs 131ns 506ns 0.0596 0 0 4.49 KB
#3205 EnrichedLog net472 153μs 242ns 939ns 0.682 0.227 0 4.67 KB
#3205 EnrichedLog netcoreapp3.1 115μs 127ns 475ns 0 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.96μs 3.16ns 12.2ns 0.568 0.00297 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.48μs 2.72ns 10.2ns 0.053 0 0 3.91 KB
#3205 EnrichedLog net472 5.62μs 9.87ns 38.2ns 0.569 0.0028 0 3.59 KB
#3205 EnrichedLog netcoreapp3.1 4.28μs 7.84ns 30.4ns 0.0527 0.00211 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.35μs 3.64ns 13.6ns 0.217 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.85μs 1.06ns 4.1ns 0.0184 0 0 1.32 KB
#3205 SendReceive net472 2.43μs 2.29ns 8.25ns 0.217 0 0 1.37 KB
#3205 SendReceive netcoreapp3.1 1.79μs 1.88ns 7.29ns 0.0179 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.14μs 2.41ns 9.35ns 0.354 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.37μs 2.88ns 11.2ns 0.024 0 0 1.8 KB
#3205 EnrichedLog net472 4.96μs 2.99ns 11.6ns 0.354 0 0 2.23 KB
#3205 EnrichedLog netcoreapp3.1 4.49μs 2.35ns 8.15ns 0.0224 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.18μs 0.559ns 2.17ns 0.128 0 0 810 B
master StartFinishSpan netcoreapp3.1 949ns 0.417ns 1.56ns 0.0104 0 0 760 B
master StartFinishScope net472 1.49μs 0.667ns 2.58ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.06μs 0.331ns 1.24ns 0.0117 0 0 880 B
#3205 StartFinishSpan net472 1.18μs 0.392ns 1.47ns 0.128 0 0 810 B
#3205 StartFinishSpan netcoreapp3.1 931ns 0.492ns 1.9ns 0.0103 0 0 760 B
#3205 StartFinishScope net472 1.43μs 0.738ns 2.86ns 0.141 0 0 891 B
#3205 StartFinishScope netcoreapp3.1 1.11μs 0.466ns 1.8ns 0.0117 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.51μs 0.899ns 3.48ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.17μs 0.528ns 1.98ns 0.0117 0 0 880 B
#3205 RunOnMethodBegin net472 1.53μs 0.525ns 2.03ns 0.141 0 0 891 B
#3205 RunOnMethodBegin netcoreapp3.1 1.22μs 0.602ns 2.33ns 0.0115 0 0 880 B

@lucaspimentel
lucaspimentel merged commit f8aa12c into master Sep 15, 2022
@lucaspimentel
lucaspimentel deleted the lpimentel/string-diffs-in-tests branch September 15, 2022 15:12
@github-actions github-actions Bot added this to the vNext milestone Sep 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@andrewlock andrewlock andrewlock approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:tests unit tests, integration tests type:enhancement Improvement to an existing feature

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@lucaspimentel @andrewlock @pierotibou