Skip to content

Schedule CI to run 10 times on a Saturday morning#3202

Merged
andrewlock merged 2 commits into
masterfrom
andrew/schedule-10-master-builds
Sep 14, 2022
Merged

Schedule CI to run 10 times on a Saturday morning#3202
andrewlock merged 2 commits into
masterfrom
andrew/schedule-10-master-builds

Conversation

@andrewlock

@andrewlock andrewlock commented Sep 12, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

Summary of changes

  • Add an AzDo schedule that runs CI 10 times on a Saturday morning
  • Always run the exploration tests on these builds

Reason for change

We're struggling with flaky tests at the moment, so we want more data about what's failing.

Implementation details

Setup a schedule to start a CI run on master, every 6mins, starting at 1am on Saturday. That gives us 10 runs.

Test coverage

We can find out if it worked in a week!

Other details

Crontab expression reference, AzDo schedule docs

@andrewlock andrewlock added the area:builds project files, build scripts, pipelines, versioning, releases, packages label Sep 12, 2022
@andrewlock
andrewlock requested review from a team as code owners September 12, 2022 09:02
OmerRaviv
OmerRaviv approved these changes Sep 12, 2022
View reviewed changes

@OmerRaviv OmerRaviv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

pierotibou
pierotibou approved these changes Sep 12, 2022
View reviewed changes
Comment thread .azure-pipelines/ultimate-pipeline.yml
always: true

- cron: "*/6 1 * * Sat"
displayName: Saturday CI, every 6 minutes from 1am

@pierotibou pierotibou Sep 12, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about the polar bears ? :)

@andrewlock andrewlock Sep 12, 2022

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we're running the VMs either way, may as well use them for something 😉

@andrewlock

This comment has been minimized.

Sign in to view
@andrewlock
andrewlock force-pushed the andrew/schedule-10-master-builds branch from 4e71093 to 5ff1394 Compare September 12, 2022 13:18
@andrewlock

andrewlock commented Sep 12, 2022

Copy link
Copy Markdown
Member Author

Benchmarks Report 🐌

Benchmarks for #3202 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.243
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 715μs 547ns 2.12μs 0.359 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 472μs 177ns 663ns 0 0 0 2.59 KB
#3202 WriteAndFlushEnrichedTraces net472 717μs 589ns 2.28μs 0.357 0 0 3.18 KB
#3202 WriteAndFlushEnrichedTraces netcoreapp3.1 457μs 387ns 1.5μs 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #3202

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.243 185.35 230.38

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 186ns 0.242ns 0.938ns 0.0676 0 0 425 B
master AllCycleSimpleBody netcoreapp3.1 236ns 0.328ns 1.27ns 0.00576 0 0 424 B
master AllCycleMoreComplexBody net472 187ns 0.163ns 0.587ns 0.0638 0 0 401 B
master AllCycleMoreComplexBody netcoreapp3.1 232ns 0.204ns 0.791ns 0.00548 0 0 400 B
master BodyExtractorSimpleBody net472 260ns 0.212ns 0.821ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 224ns 0.169ns 0.634ns 0.00363 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.7μs 12.3ns 44.4ns 1.2 0.0147 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 12.1μs 20.7ns 77.5ns 0.0896 0 0 6.75 KB
#3202 AllCycleSimpleBody net472 230ns 0.207ns 0.801ns 0.0676 0 0 425 B
#3202 AllCycleSimpleBody netcoreapp3.1 237ns 0.27ns 1.05ns 0.00589 0 0 424 B
#3202 AllCycleMoreComplexBody net472 185ns 0.18ns 0.699ns 0.0637 0 0 401 B
#3202 AllCycleMoreComplexBody netcoreapp3.1 233ns 0.406ns 1.62ns 0.00551 0 0 400 B
#3202 BodyExtractorSimpleBody net472 258ns 0.205ns 0.795ns 0.0573 0 0 361 B
#3202 BodyExtractorSimpleBody netcoreapp3.1 224ns 0.221ns 0.767ns 0.00372 0 0 272 B
#3202 BodyExtractorMoreComplexBody net472 14.8μs 10.6ns 41.1ns 1.21 0.0148 0 7.62 KB
#3202 BodyExtractorMoreComplexBody netcoreapp3.1 12.1μs 15.2ns 59ns 0.0911 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown 🤷 Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 182μs 223ns 864ns 0.273 0 0 20.57 KB
#3202 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3202 SendRequest netcoreapp3.1 182μs 437ns 1.69μs 0.272 0.0907 0 20.58 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.85μs 1.21ns 4.67ns 0.15 0.000919 0 947 B
master ExecuteNonQuery netcoreapp3.1 1.35μs 0.576ns 2.15ns 0.013 0 0 936 B
#3202 ExecuteNonQuery net472 1.89μs 2.14ns 8.27ns 0.15 0.000946 0 947 B
#3202 ExecuteNonQuery netcoreapp3.1 1.43μs 1.55ns 5.99ns 0.0126 0.00074 0 936 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.54μs 0.905ns 3.5ns 0.183 0 0 1.16 KB
master CallElasticsearch netcoreapp3.1 1.55μs 0.483ns 1.81ns 0.0146 0 0 1.1 KB
master CallElasticsearchAsync net472 2.59μs 0.533ns 2.07ns 0.205 0 0 1.29 KB
master CallElasticsearchAsync netcoreapp3.1 1.7μs 0.476ns 1.78ns 0.0161 0 0 1.22 KB
#3202 CallElasticsearch net472 2.59μs 1.93ns 7.24ns 0.183 0 0 1.16 KB
#3202 CallElasticsearch netcoreapp3.1 1.57μs 0.573ns 2.06ns 0.0141 0 0 1.1 KB
#3202 CallElasticsearchAsync net472 2.78μs 1.57ns 5.67ns 0.205 0 0 1.29 KB
#3202 CallElasticsearchAsync netcoreapp3.1 1.59μs 1.02ns 3.69ns 0.0158 0 0 1.22 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.64μs 4.19ns 15.7ns 0.226 0 0 1.42 KB
master ExecuteAsync netcoreapp3.1 1.71μs 2.64ns 10.2ns 0.0179 0 0 1.34 KB
#3202 ExecuteAsync net472 2.61μs 7.08ns 27.4ns 0.225 0 0 1.42 KB
#3202 ExecuteAsync netcoreapp3.1 1.73μs 3.17ns 11.8ns 0.018 0 0 1.34 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.69μs 13.6ns 52.6ns 0.438 0 0 2.77 KB
master SendAsync netcoreapp3.1 3.6μs 8.06ns 31.2ns 0.0359 0 0 2.6 KB
#3202 SendAsync net472 5.66μs 12.8ns 49.4ns 0.438 0 0 2.77 KB
#3202 SendAsync netcoreapp3.1 3.61μs 11.6ns 44.9ns 0.034 0 0 2.6 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 3.21μs 1.47ns 5.67ns 0.287 0 0 1.81 KB
master EnrichedLog netcoreapp3.1 2.45μs 0.701ns 2.62ns 0.0258 0 0 1.85 KB
#3202 EnrichedLog net472 3.24μs 3.57ns 13.8ns 0.287 0 0 1.81 KB
#3202 EnrichedLog netcoreapp3.1 2.57μs 1.22ns 4.4ns 0.0243 0 0 1.85 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 151μs 47.4ns 184ns 0.678 0.226 0 4.65 KB
master EnrichedLog netcoreapp3.1 115μs 75.7ns 293ns 0.0574 0 0 4.49 KB
#3202 EnrichedLog net472 151μs 217ns 842ns 0.678 0.226 0 4.66 KB
#3202 EnrichedLog netcoreapp3.1 116μs 143ns 536ns 0.0593 0 0 4.49 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.55μs 9.07ns 32.7ns 0.569 0.00277 0 3.59 KB
master EnrichedLog netcoreapp3.1 4.26μs 6.92ns 25.9ns 0.0527 0 0 3.91 KB
#3202 EnrichedLog net472 5.59μs 10.7ns 41.4ns 0.569 0.00279 0 3.59 KB
#3202 EnrichedLog netcoreapp3.1 4.27μs 5.52ns 19.9ns 0.0525 0 0 3.91 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.35μs 1.92ns 7.17ns 0.217 0 0 1.37 KB
master SendReceive netcoreapp3.1 1.87μs 2.08ns 7.78ns 0.0179 0 0 1.32 KB
#3202 SendReceive net472 2.36μs 1.8ns 6.72ns 0.218 0 0 1.37 KB
#3202 SendReceive netcoreapp3.1 1.8μs 1.01ns 3.93ns 0.018 0 0 1.32 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.03μs 2.5ns 9.67ns 0.352 0 0 2.23 KB
master EnrichedLog netcoreapp3.1 4.35μs 1.79ns 6.7ns 0.0239 0 0 1.8 KB
#3202 EnrichedLog net472 5.16μs 3.43ns 13.3ns 0.353 0 0 2.23 KB
#3202 EnrichedLog netcoreapp3.1 4.43μs 2.11ns 8.15ns 0.0245 0 0 1.8 KB
Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 1.12μs 0.793ns 2.97ns 0.128 0 0 810 B
master StartFinishSpan netcoreapp3.1 997ns 0.211ns 0.791ns 0.0102 0 0 760 B
master StartFinishScope net472 1.42μs 0.546ns 2.04ns 0.141 0 0 891 B
master StartFinishScope netcoreapp3.1 1.08μs 0.76ns 2.94ns 0.0118 0 0 880 B
#3202 StartFinishSpan net472 1.19μs 0.762ns 2.85ns 0.128 0 0 810 B
#3202 StartFinishSpan netcoreapp3.1 925ns 0.905ns 3.39ns 0.0102 0 0 760 B
#3202 StartFinishScope net472 1.4μs 0.793ns 3.07ns 0.141 0 0 891 B
#3202 StartFinishScope netcoreapp3.1 1.05μs 0.331ns 1.28ns 0.0121 0 0 880 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.51μs 1.17ns 4.53ns 0.141 0 0 891 B
master RunOnMethodBegin netcoreapp3.1 1.12μs 0.527ns 1.97ns 0.0118 0 0 880 B
#3202 RunOnMethodBegin net472 1.56μs 0.989ns 3.83ns 0.141 0 0 891 B
#3202 RunOnMethodBegin netcoreapp3.1 1.23μs 0.67ns 2.6ns 0.0117 0 0 880 B

@andrewlock

andrewlock commented Sep 13, 2022

Copy link
Copy Markdown
Member Author

Code Coverage Report 📊

✔️ Merging #3202 into master will not change line coverage
✔️ Merging #3202 into master will not change branch coverage
✔️ Merging #3202 into master will not change complexity

master #3202 Change
Lines 17971 / 24517 17936 / 24517
Lines % 73% 73% 0% ✔️
Branches 10585 / 15050 10582 / 15050
Branches % 70% 70% 0% ✔️
Complexity 16261 16261 0 ✔️

View the full report for further details:

Datadog.Trace Breakdown ✔️

master #3202 Change
Lines % 73% 73% 0% ✔️
Branches % 70% 70% 0% ✔️
Complexity 16261 16261 0 ✔️

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.GitInfo -17% ⛔ -11% ⛔ 0 ✔️
Datadog.Trace.Ci.CIVisibility 5% ✔️ 6% ✔️ 0 ✔️

View the full reports for further details:

@andrewlock
andrewlock merged commit 5554200 into master Sep 14, 2022
@andrewlock
andrewlock deleted the andrew/schedule-10-master-builds branch September 14, 2022 08:15
@github-actions github-actions Bot added this to the vNext milestone Sep 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@OmerRaviv OmerRaviv OmerRaviv approved these changes
@pierotibou pierotibou pierotibou approved these changes

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages

Projects

None yet

Milestone

2.15.0

Development

Successfully merging this pull request may close these issues.

3 participants

@andrewlock @OmerRaviv @pierotibou