Skip to content

Ensure we always run exploration tests on master#3243

Merged
andrewlock merged 3 commits into
masterfrom
andrew/fix-exploration-tests
Sep 22, 2022
Merged

Ensure we always run exploration tests on master#3243
andrewlock merged 3 commits into
masterfrom
andrew/fix-exploration-tests

Conversation

@andrewlock

@andrewlock andrewlock commented Sep 21, 2022

Copy link
Copy Markdown
Member

Summary of changes

Updates the exploration tests variable generation to check if running on master - if so, always run exploration tests

Reason for change

We were triggering the exploration tests to run on master on the scheduled build, but not actually generating the variables

Implementation details

  • Have the exploration test build check if running on master - if so, run all checks
  • Removed the (now unnecessary) isScheduledBuildOnMain variable

Test coverage

YOLO

Other details

This is a slight behaviour change, as we now always run the exploration tests on master, but I think that's a reasonable update.

@andrewlock andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests labels Sep 21, 2022
@andrewlock
andrewlock requested review from a team as code owners September 21, 2022 14:25
@andrewlock

This comment has been minimized.

Sign in to view
OmerRaviv
OmerRaviv approved these changes Sep 21, 2022
View reviewed changes

@OmerRaviv OmerRaviv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andrewlock

andrewlock commented Sep 21, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3243 compared to master:

  • All benchmarks have the same speed
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 710μs 506ns 1.96μs 0.357 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 459μs 275ns 1.06μs 0 0 0 2.58 KB
#3243 WriteAndFlushEnrichedTraces net472 716μs 612ns 2.37μs 0.355 0 0 3.18 KB
#3243 WriteAndFlushEnrichedTraces netcoreapp3.1 454μs 102ns 396ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 182ns 0.155ns 0.599ns 0.0675 9.18E-05 0 425 B
master AllCycleSimpleBody netcoreapp3.1 239ns 0.154ns 0.575ns 0.00577 0 0 424 B
master AllCycleMoreComplexBody net472 182ns 0.179ns 0.67ns 0.0637 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 245ns 0.304ns 1.18ns 0.00557 0 0 400 B
master BodyExtractorSimpleBody net472 255ns 0.209ns 0.783ns 0.0574 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 216ns 0.251ns 0.938ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.8μs 9.99ns 36ns 1.21 0.0148 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12μs 18.8ns 72.7ns 0.09 0 0 6.75 KB
#3243 AllCycleSimpleBody net472 201ns 0.378ns 1.46ns 0.0675 0 0 425 B
#3243 AllCycleSimpleBody netcoreapp3.1 245ns 0.458ns 1.77ns 0.00579 0 0 424 B
#3243 AllCycleMoreComplexBody net472 190ns 0.238ns 0.891ns 0.0637 0 0 401 B
#3243 AllCycleMoreComplexBody netcoreapp3.1 259ns 0.406ns 1.57ns 0.00546 0 0 400 B
#3243 BodyExtractorSimpleBody net472 263ns 0.476ns 1.84ns 0.0573 0 0 361 B
#3243 BodyExtractorSimpleBody netcoreapp3.1 230ns 0.294ns 1.14ns 0.00369 0 0 272 B
#3243 BodyExtractorMoreComplexBody net472 15.5μs 26.3ns 102ns 1.21 0.0154 0 7.62 KB
#3243 BodyExtractorMoreComplexBody netcoreapp3.1 12.4μs 16.7ns 62.4ns 0.0927 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 178μs 163ns 588ns 0.268 0 0 20.58 KB
#3243 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3243 SendRequest netcoreapp3.1 183μs 368ns 1.43μs 0.271 0 0 20.6 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.84μs 1.08ns 4.17ns 0.15 0 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.55μs 1.26ns 4.87ns 0.0129 0 0 936 B
#3243 ExecuteNonQuery net472 1.83μs 0.792ns 2.96ns 0.15 0.000908 0 947 B
#3243 ExecuteNonQuery netcoreapp3.1 1.4μs 0.777ns 2.91ns 0.0126 0 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.5μs 1.45ns 5.43ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.56μs 0.979ns 3.66ns 0.0148 0 0 1.1 KB
master CallElasticsearchAsync net472 2.59μs 1.71ns 6.61ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.69μs 0.671ns 2.42ns 0.0162 0 0 1.22 KB
#3243 CallElasticsearch net472 2.57μs 1.67ns 6.25ns 0.184 0 0 1.16 KB
#3243 CallElasticsearch netcoreapp3.1 1.47μs 0.724ns 2.71ns 0.0148 0 0 1.1 KB
#3243 CallElasticsearchAsync net472 2.68μs 2.15ns 8.34ns 0.205 0 0 1.29 KB
#3243 CallElasticsearchAsync netcoreapp3.1 1.66μs 0.383ns 1.43ns 0.0166 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.82μs 11ns 42.7ns 0.225 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.71μs 1.27ns 4.91ns 0.0179 0 0 1.34 KB
#3243 ExecuteAsync net472 2.84μs 1.61ns 6.02ns 0.225 0 0 1.42 KB
#3243 ExecuteAsync netcoreapp3.1 1.75μs 0.469ns 1.75ns 0.0175 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.85μs 3.31ns 12.8ns 0.438 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.66μs 3.8ns 14.2ns 0.0349 0 0 2.6 KB
#3243 SendAsync net472 5.82μs 2.53ns 9.45ns 0.44 0 0 2.77 KB
#3243 SendAsync netcoreapp3.1 3.67μs 3.78ns 14.7ns 0.0352 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.24μs 3.36ns 13ns 0.288 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.52μs 1.48ns 5.74ns 0.0252 0 0 1.85 KB
#3243 EnrichedLog net472 3.24μs 2.49ns 9.33ns 0.287 0.0016 0 1.81 KB
#3243 EnrichedLog netcoreapp3.1 2.49μs 1.42ns 5.33ns 0.0249 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 152μs 155ns 600ns 0.685 0.228 0 4.65 KB
master EnrichedLog netcoreapp3.1 118μs 213ns 823ns 0.0579 0 0 4.49 KB
#3243 EnrichedLog net472 155μs 218ns 843ns 0.688 0.229 0 4.65 KB
#3243 EnrichedLog netcoreapp3.1 119μs 160ns 621ns 0.0582 0 0 4.5 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.72μs 14ns 54.3ns 0.57 0.00285 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.39μs 3.4ns 12.3ns 0.0532 0.00213 0 3.91 KB
#3243 EnrichedLog net472 5.89μs 1.57ns 5.66ns 0.569 0.00293 0 3.59 KB
#3243 EnrichedLog netcoreapp3.1 4.46μs 3.13ns 11.7ns 0.0527 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.38μs 3.65ns 14.1ns 0.218 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.76μs 0.62ns 2.4ns 0.0185 0 0 1.32 KB
#3243 SendReceive net472 2.35μs 2.42ns 9.36ns 0.218 0 0 1.37 KB
#3243 SendReceive netcoreapp3.1 1.78μs 1.45ns 5.62ns 0.0178 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.95μs 1.44ns 4.97ns 0.353 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.3μs 3.04ns 11.4ns 0.0236 0 0 1.8 KB
#3243 EnrichedLog net472 4.95μs 1.56ns 5.84ns 0.354 0.00246 0 2.23 KB
#3243 EnrichedLog netcoreapp3.1 4.36μs 1.64ns 6.35ns 0.0238 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.15μs 0.711ns 2.75ns 0.129 0 0 810 B
master StartFinishSpan netcoreapp3.1 985ns 0.614ns 2.38ns 0.0103 0 0 760 B
master StartFinishScope net472 1.45μs 0.422ns 1.52ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.05μs 0.371ns 1.44ns 0.0116 0 0 880 B
#3243 StartFinishSpan net472 1.19μs 0.455ns 1.76ns 0.128 0 0 810 B
#3243 StartFinishSpan netcoreapp3.1 923ns 0.604ns 2.34ns 0.0101 0 0 760 B
#3243 StartFinishScope net472 1.41μs 0.869ns 3.36ns 0.141 0 0 891 B
#3243 StartFinishScope netcoreapp3.1 1.1μs 0.48ns 1.86ns 0.0121 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.58μs 1.19ns 4.62ns 0.142 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.18μs 0.571ns 2.21ns 0.0118 0 0 880 B
#3243 RunOnMethodBegin net472 1.46μs 1.44ns 5.59ns 0.141 0 0 891 B
#3243 RunOnMethodBegin netcoreapp3.1 1.17μs 0.613ns 2.21ns 0.0123 0 0 880 B

@andrewlock

andrewlock commented Sep 21, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

⛔ Merging #3243 into master will will decrease line coverage by 5%
⚠️ Merging #3243 into master will will decrease branch coverage by 3%
⛔ Merging #3243 into master will will increase complexity by 20

master #3243 Change
Lines 17963 / 24889 16841 / 24930
Lines % 72% 68% -5% ⛔
Branches 10472 / 15254 10033 / 15274
Branches % 69% 66% -3% ⚠️
Complexity 16477 16497 20 ⛔

View the full report for further details:

Datadog.Trace Breakdown ⛔

master #3243 Change
Lines % 72% 68% -5% ⛔
Branches % 69% 66% -3% ⚠️
Complexity 16477 16497 20 ⛔

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Debugger.PInvoke.NativeMethodProbeDefinition -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Sink.SnapshotSink -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Snapshots.LazySnapshotSerializerFieldsAndPropsSelector -100% ⛔ -100% ⛔ 0 ✔️
Datadog.Trace.Debugger.Snapshots.DebuggerSnapshotCreator -100% ⛔ -68% ⛔ 1 ⛔
BoundLineProbeLocation -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.CapturedLines -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.Instrumentation.DebuggerReturn -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.Instrumentation.MethodMetadataInfo -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.Models.LineProbeResolveResult -100% ⛔ 0% ✔️ 0 ✔️
Datadog.Trace.Debugger.PInvoke.NativeLineProbeDefinition -100% ⛔ 0% ✔️ 0 ✔️
...And 48 more

View the full reports for further details:

zacharycmontoya
zacharycmontoya reviewed Sep 21, 2022
View reviewed changes
Comment thread tracer/build/_build/Build.VariableGenerations.cs

static bool IsGitBaseBranch(string baseBranch)
=> string.Equals(
GitTasks.Git("rev-parse --abbrev-ref HEAD").First().Text,

@zacharycmontoya zacharycmontoya Sep 21, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You really are the git whisperer ✨

@andrewlock
andrewlock merged commit 88d1476 into master Sep 22, 2022
@andrewlock
andrewlock deleted the andrew/fix-exploration-tests branch September 22, 2022 08:44
@github-actions github-actions Bot added this to the vNext milestone Sep 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@OmerRaviv OmerRaviv OmerRaviv approved these changes
@zacharycmontoya zacharycmontoya zacharycmontoya approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

4 participants

@andrewlock @OmerRaviv @zacharycmontoya @pierotibou