Skip to content

Ensure we clean log files before testing with Nuke#5950

Merged
andrewlock merged 1 commit into
masterfrom
andrew/ci/ensure-clean-dir
Aug 27, 2024
Merged

Ensure we clean log files before testing with Nuke#5950
andrewlock merged 1 commit into
masterfrom
andrew/ci/ensure-clean-dir

Conversation

@andrewlock

@andrewlock andrewlock commented Aug 27, 2024

Copy link
Copy Markdown
Member

Summary of changes

Cleans the log file directory before running tests

Reason for change

Some tests assert on the presence/absence of log files. Tests should handle cleaning up before they run, but this is a stop gap.

Implementation details

Clean the test logs directory before running the tests

Test coverage

This is the test

@andrewlock andrewlock added area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests labels Aug 27, 2024
@andrewlock
andrewlock requested a review from a team as a code owner August 27, 2024 11:11
@datadog-ddstaging

datadog-ddstaging Bot commented Aug 27, 2024 •
edited
Loading

Copy link
Copy Markdown

Datadog Report

Branch report: andrew/ci/ensure-clean-dir
Commit report: 742e003
Test service: dd-trace-dotnet

✅ 0 Failed, 320871 Passed, 2715 Skipped, 30h 36m 57.47s Total Time
❄️ 1 New Flaky

New Flaky Tests (1)

  • AzureFunctionsTests+InProcessRuntimeV4.SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests - Last Failure

    Expand for error 
     Expected collection to contain at least 21 item(s) because we want to ensure that we don't timeout while waiting for spans from the mock tracer agent, but found 2: {{TraceId: 4694888903721452492, SpanId: 9354509548256486078, Name: http.request, Resource: GET localhost:7071/api/trigger, Service: AzureFunctionsAllTriggers-http-client}, {TraceId: 4694888903721452492, SpanId: 17599778338168234800, Name: azure-functions.invoke, Resource: Timer TriggerAllTimer, Service: AzureFunctionsAllTriggers}}.

@andrewlock

andrewlock commented Aug 27, 2024

Copy link
Copy Markdown
Member Author

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

@kevingosse kevingosse mentioned this pull request Aug 27, 2024
Merged
@andrewlock

andrewlock commented Aug 27, 2024

Copy link
Copy Markdown
Member Author

Benchmarks Report for tracer 🐌

Benchmarks for #5950 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.257
  • 1 benchmarks are slower, with geometric mean 1.297
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.68μs 42.5ns 272ns 0.0148 0.00738 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.91μs 52.5ns 273ns 0.0192 0.00962 0 5.62 KB
master StartStopWithChild net472 16μs 55.9ns 216ns 1.02 0.302 0.103 6.06 KB
#5950 StartStopWithChild net6.0 7.97μs 45.6ns 329ns 0.0198 0.00791 0 5.42 KB
#5950 StartStopWithChild netcoreapp3.1 10.4μs 58.5ns 405ns 0.03 0.015 0 5.62 KB
#5950 StartStopWithChild net472 17.5μs 91.9ns 441ns 1.01 0.299 0.0881 6.07 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 465μs 200ns 722ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 634μs 278ns 1.08μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 836μs 677ns 2.62μs 0.417 0 0 3.3 KB
#5950 WriteAndFlushEnrichedTraces net6.0 462μs 288ns 1.04μs 0 0 0 2.7 KB
#5950 WriteAndFlushEnrichedTraces netcoreapp3.1 622μs 214ns 828ns 0 0 0 2.7 KB
#5950 WriteAndFlushEnrichedTraces net472 845μs 598ns 2.32μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 192μs 1.11μs 9.5μs 0.182 0 0 18.45 KB
master SendRequest netcoreapp3.1 208μs 1.15μs 7.46μs 0.198 0 0 20.61 KB
master SendRequest net472 0.001ns 0.000448ns 0.00168ns 0 0 0 0 b
#5950 SendRequest net6.0 189μs 1.03μs 6.18μs 0.182 0 0 18.45 KB
#5950 SendRequest netcoreapp3.1 213μs 1.28μs 12.6μs 0.207 0 0 20.61 KB
#5950 SendRequest net472 0.00219ns 0.000747ns 0.0028ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #5950

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.73 KB 41.95 KB 217 B 0.52%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 555μs 2.33μs 8.42μs 0.566 0 0 41.57 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 669μs 2.33μs 8.41μs 0.326 0 0 41.73 KB
master WriteAndFlushEnrichedTraces net472 852μs 2.35μs 9.1μs 8.39 2.52 0.419 53.3 KB
#5950 WriteAndFlushEnrichedTraces net6.0 554μs 1.78μs 6.89μs 0.558 0 0 41.71 KB
#5950 WriteAndFlushEnrichedTraces netcoreapp3.1 686μs 2.89μs 11.2μs 0.334 0 0 41.95 KB
#5950 WriteAndFlushEnrichedTraces net472 887μs 3.55μs 13.8μs 8.42 2.66 0.443 53.29 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.23μs 1.01ns 3.8ns 0.0146 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.73μs 1.47ns 5.71ns 0.0137 0 0 1.02 KB
master ExecuteNonQuery net472 1.98μs 1.89ns 7.31ns 0.157 0 0 987 B
#5950 ExecuteNonQuery net6.0 1.36μs 0.783ns 2.82ns 0.0144 0 0 1.02 KB
#5950 ExecuteNonQuery netcoreapp3.1 1.67μs 1.56ns 6.05ns 0.0133 0 0 1.02 KB
#5950 ExecuteNonQuery net472 1.98μs 3.63ns 14.1ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.23μs 0.999ns 3.87ns 0.0135 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.63μs 0.555ns 2.15ns 0.013 0 0 976 B
master CallElasticsearch net472 2.43μs 1.72ns 6.66ns 0.157 0 0 995 B
master CallElasticsearchAsync net6.0 1.38μs 1.31ns 4.91ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.57μs 1.27ns 4.76ns 0.014 0 0 1.02 KB
master CallElasticsearchAsync net472 2.58μs 1.71ns 6.42ns 0.167 0 0 1.05 KB
#5950 CallElasticsearch net6.0 1.11μs 3.77ns 14.1ns 0.0138 0 0 976 B
#5950 CallElasticsearch netcoreapp3.1 1.56μs 0.763ns 2.86ns 0.0132 0 0 976 B
#5950 CallElasticsearch net472 2.48μs 1.38ns 5.15ns 0.158 0.00123 0 995 B
#5950 CallElasticsearchAsync net6.0 1.26μs 0.671ns 2.6ns 0.0133 0 0 952 B
#5950 CallElasticsearchAsync netcoreapp3.1 1.56μs 2.64ns 10.2ns 0.0132 0 0 1.02 KB
#5950 CallElasticsearchAsync net472 2.59μs 1.72ns 6.67ns 0.166 0.00129 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.26μs 0.661ns 2.47ns 0.0133 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.55μs 0.555ns 2ns 0.0131 0 0 952 B
master ExecuteAsync net472 1.78μs 1.47ns 5.71ns 0.145 0 0 915 B
#5950 ExecuteAsync net6.0 1.35μs 1.78ns 6.88ns 0.0134 0 0 952 B
#5950 ExecuteAsync netcoreapp3.1 1.65μs 0.4ns 1.44ns 0.0125 0 0 952 B
#5950 ExecuteAsync net472 1.7μs 0.624ns 2.33ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.21μs 2.09ns 8.11ns 0.0315 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.09μs 2.27ns 8.81ns 0.0358 0 0 2.76 KB
master SendAsync net472 7.75μs 3.24ns 12.6ns 0.499 0 0 3.15 KB
#5950 SendAsync net6.0 4.09μs 1.86ns 6.95ns 0.0308 0 0 2.22 KB
#5950 SendAsync netcoreapp3.1 5.06μs 2.27ns 7.85ns 0.0355 0 0 2.76 KB
#5950 SendAsync net472 7.69μs 1.9ns 7.36ns 0.499 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.48μs 0.708ns 2.65ns 0.023 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.38μs 1.1ns 4.12ns 0.0217 0 0 1.64 KB
master EnrichedLog net472 2.68μs 1.93ns 7.24ns 0.249 0 0 1.57 KB
#5950 EnrichedLog net6.0 1.56μs 0.588ns 2.2ns 0.0227 0 0 1.64 KB
#5950 EnrichedLog netcoreapp3.1 2.28μs 0.742ns 2.78ns 0.0219 0 0 1.64 KB
#5950 EnrichedLog net472 2.7μs 1.34ns 5.17ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 116μs 166ns 621ns 0.0578 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 119μs 234ns 905ns 0 0 0 4.28 KB
master EnrichedLog net472 148μs 225ns 872ns 0.668 0.223 0 4.46 KB
#5950 EnrichedLog net6.0 115μs 291ns 1.13μs 0.0576 0 0 4.28 KB
#5950 EnrichedLog netcoreapp3.1 121μs 175ns 654ns 0 0 0 4.28 KB
#5950 EnrichedLog net472 149μs 221ns 858ns 0.672 0.224 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.95μs 1.62ns 6.26ns 0.0313 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.09μs 1.07ns 4.01ns 0.0286 0 0 2.2 KB
master EnrichedLog net472 4.78μs 2.91ns 11.3ns 0.319 0 0 2.02 KB
#5950 EnrichedLog net6.0 3.14μs 1.18ns 4.58ns 0.0298 0 0 2.2 KB
#5950 EnrichedLog netcoreapp3.1 4.1μs 1.88ns 6.52ns 0.0299 0 0 2.2 KB
#5950 EnrichedLog net472 4.82μs 1.63ns 6.33ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.37μs 0.604ns 2.34ns 0.0164 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.76μs 1.38ns 5.35ns 0.0149 0 0 1.14 KB
master SendReceive net472 2.18μs 1.64ns 5.9ns 0.184 0.00109 0 1.16 KB
#5950 SendReceive net6.0 1.35μs 0.62ns 2.32ns 0.0162 0 0 1.14 KB
#5950 SendReceive netcoreapp3.1 1.73μs 1.46ns 5.46ns 0.0157 0 0 1.14 KB
#5950 SendReceive net472 2.21μs 1.33ns 4.98ns 0.183 0.0011 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.81μs 0.997ns 3.86ns 0.0225 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.92μs 1.92ns 7.43ns 0.0217 0 0 1.65 KB
master EnrichedLog net472 4.36μs 2.83ns 11ns 0.324 0 0 2.04 KB
#5950 EnrichedLog net6.0 2.77μs 0.858ns 3.21ns 0.0221 0 0 1.6 KB
#5950 EnrichedLog netcoreapp3.1 3.83μs 1.74ns 6.73ns 0.021 0 0 1.65 KB
#5950 EnrichedLog net472 4.48μs 2.79ns 10.8ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #5950

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.297 396.03 513.51

Faster 🎉 in #5950

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.257 605.08 481.23

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 396ns 0.124ns 0.463ns 0.00819 0 0 576 B
master StartFinishSpan netcoreapp3.1 606ns 0.414ns 1.6ns 0.00788 0 0 576 B
master StartFinishSpan net472 630ns 0.601ns 2.33ns 0.0918 0 0 578 B
master StartFinishScope net6.0 605ns 0.291ns 1.13ns 0.0097 0 0 696 B
master StartFinishScope netcoreapp3.1 717ns 0.508ns 1.97ns 0.00972 0 0 696 B
master StartFinishScope net472 841ns 0.845ns 3.27ns 0.104 0 0 658 B
#5950 StartFinishSpan net6.0 515ns 1.44ns 5.58ns 0.00797 0 0 576 B
#5950 StartFinishSpan netcoreapp3.1 623ns 1.17ns 4.53ns 0.00781 0 0 576 B
#5950 StartFinishSpan net472 575ns 0.252ns 0.977ns 0.0918 0 0 578 B
#5950 StartFinishScope net6.0 481ns 0.914ns 3.42ns 0.00978 0 0 696 B
#5950 StartFinishScope netcoreapp3.1 730ns 0.233ns 0.902ns 0.00921 0 0 696 B
#5950 StartFinishScope net472 849ns 0.48ns 1.8ns 0.105 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 579ns 0.354ns 1.37ns 0.00966 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 922ns 1.08ns 4.02ns 0.00931 0 0 696 B
master RunOnMethodBegin net472 1.05μs 0.801ns 3.1ns 0.104 0 0 658 B
#5950 RunOnMethodBegin net6.0 585ns 0.439ns 1.7ns 0.00966 0 0 696 B
#5950 RunOnMethodBegin netcoreapp3.1 989ns 0.317ns 1.14ns 0.00931 0 0 696 B
#5950 RunOnMethodBegin net472 1.12μs 0.612ns 2.37ns 0.104 0 0 658 B

@andrewlock

andrewlock commented Aug 27, 2024

Copy link
Copy Markdown
Member Author

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5950) (11.762M)   : 0, 11762180
    master (11.881M)   : 0, 11881253
    benchmarks/2.9.0 (11.711M)   : 0, 11710619

    section Automatic
    This PR (5950) (7.892M)   : 0, 7891641
    master (7.883M)   : 0, 7882754
    benchmarks/2.9.0 (8.265M)   : 0, 8264630

    section Trace stats
    master (8.068M)   : 0, 8067538

    section Manual
    master (11.802M)   : 0, 11801711

    section Manual + Automatic
    This PR (5950) (7.334M)   : 0, 7334039
    master (7.353M)   : 0, 7353425

    section DD_TRACE_ENABLED=0
    master (11.160M)   : 0, 11159995
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5950) (9.454M)   : 0, 9453745
    master (9.455M)   : 0, 9455386
    benchmarks/2.9.0 (9.717M)   : 0, 9716586

    section Automatic
    This PR (5950) (6.536M)   : 0, 6535656
    master (6.450M)   : 0, 6450497

    section Trace stats
    master (6.922M)   : 0, 6922039

    section Manual
    master (9.388M)   : 0, 9388217

    section Manual + Automatic
    This PR (5950) (6.010M)   : 0, 6010181
    master (6.039M)   : 0, 6038692

    section DD_TRACE_ENABLED=0
    master (8.958M)   : 0, 8958458
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5950) (10.095M)   : 0, 10094564

    section Automatic
    This PR (5950) (6.645M)   : 0, 6645172

    section Manual + Automatic
    This PR (5950) (6.088M)   : 0, 6088392
Loading

@andrewlock
andrewlock merged commit 4407266 into master Aug 27, 2024
@andrewlock
andrewlock deleted the andrew/ci/ensure-clean-dir branch August 27, 2024 14:48
@github-actions github-actions Bot added this to the vNext-v3 milestone Aug 27, 2024
kevingosse pushed a commit that referenced this pull request Aug 28, 2024
Log to a random file in telemetry forwarder tests (#5951)
8e40369 
## Summary of changes

Use a random file name for telemetry forwarder tests.

## Reason for change

The tests could find a file left from previous runs and succeed when
they should fail.

## Other details
In theory #5950 should
already fix the issue by cleaning the log folder, but this adds extra
safety (for instance, if running tests without Nuke).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@kevingosse kevingosse kevingosse approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:builds project files, build scripts, pipelines, versioning, releases, packages area:tests unit tests, integration tests

Projects

None yet

Milestone

3.2.0

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewlock @kevingosse