Skip to content

chore: prefix system tests env vars#6108

Merged
bouwkast merged 1 commit into
masterfrom
conti/prefix-system-tests-env-vars
Oct 2, 2024
Merged

chore: prefix system tests env vars#6108
bouwkast merged 1 commit into
masterfrom
conti/prefix-system-tests-env-vars

Conversation

@wconti27

@wconti27 wconti27 commented Oct 1, 2024

Copy link
Copy Markdown
Contributor

Summary of changes

Prefix system tests env vars to not conflict with local developer env vars

Reason for change

Implementation details

Test coverage

Other details

@wconti27
wconti27 requested a review from a team as a code owner October 1, 2024 18:49
bouwkast
bouwkast approved these changes Oct 1, 2024
View reviewed changes

@bouwkast bouwkast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@andrewlock

andrewlock commented Oct 1, 2024

Copy link
Copy Markdown
Member

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (71ms)  : 67, 75
     .   : milestone, 71,
    master - mean (71ms)  : 67, 75
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (1,110ms)  : 1092, 1127
     .   : milestone, 1110,
    master - mean (1,108ms)  : 1083, 1133
     .   : milestone, 1108,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (110ms)  : 105, 115
     .   : milestone, 110,
    master - mean (111ms)  : 105, 117
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (778ms)  : 762, 795
     .   : milestone, 778,
    master - mean (778ms)  : 757, 798
     .   : milestone, 778,
Loading 
gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (94ms)  : 90, 97
     .   : milestone, 94,
    master - mean (94ms)  : 90, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (738ms)  : 715, 761
     .   : milestone, 738,
    master - mean (732ms)  : 716, 748
     .   : milestone, 732,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (191ms)  : 187, 196
     .   : milestone, 191,
    master - mean (192ms)  : 189, 194
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (1,203ms)  : 1181, 1225
     .   : milestone, 1203,
    master - mean (1,203ms)  : 1179, 1226
     .   : milestone, 1203,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (278ms)  : 273, 282
     .   : milestone, 278,
    master - mean (277ms)  : 273, 282
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (944ms)  : 924, 965
     .   : milestone, 944,
    master - mean (943ms)  : 914, 972
     .   : milestone, 943,
Loading 
gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6108) - mean (266ms)  : 263, 270
     .   : milestone, 266,
    master - mean (267ms)  : 262, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6108) - mean (928ms)  : 906, 951
     .   : milestone, 928,
    master - mean (931ms)  : 915, 947
     .   : milestone, 931,
Loading

@datadog-ddstaging

datadog-ddstaging Bot commented Oct 1, 2024 •
edited
Loading

Copy link
Copy Markdown

Datadog Report

Branch report: conti/prefix-system-tests-env-vars
Commit report: d90a0e9
Test service: dd-trace-dotnet

❌ 3 Failed (0 Known Flaky), 364565 Passed, 2071 Skipped, 16h 26m 50.19s Total Time
❄️ 1 New Flaky

❌ Failed Tests (3)

  • CallElasticsearch - Benchmarks.Trace.ElasticsearchBenchmark - Details

  • CallElasticsearchAsync - Benchmarks.Trace.ElasticsearchBenchmark - Details

  • EnrichedLog - Benchmarks.Trace.SerilogBenchmark - Details

New Flaky Tests (1)

  • HttpClient_SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.HttpMessageHandlerTests - Last Failure

    Expand for error 
     
 Expected var integration = latestIntegrations {empty} to contain key "CurlHandler".), but no such item was found.t found False.

@andrewlock

andrewlock commented Oct 1, 2024

Copy link
Copy Markdown
Member

Throughput/Crank Report ⚡

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6108) (11.250M)   : 0, 11250235
    master (11.113M)   : 0, 11112766
    benchmarks/2.9.0 (11.081M)   : 0, 11080577

    section Automatic
    This PR (6108) (7.270M)   : 0, 7269943
    master (7.264M)   : 0, 7264395
    benchmarks/2.9.0 (7.732M)   : 0, 7732233

    section Trace stats
    master (7.607M)   : 0, 7607314

    section Manual
    master (10.797M)   : 0, 10797413

    section Manual + Automatic
    This PR (6108) (6.705M)   : 0, 6704659
    master (6.861M)   : 0, 6860788

    section DD_TRACE_ENABLED=0
    master (10.026M)   : 0, 10025656
Loading 
gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6108) (9.837M)   : 0, 9837233
    master (9.642M)   : 0, 9642367
    benchmarks/2.9.0 (9.798M)   : 0, 9798067

    section Automatic
    This PR (6108) (6.507M)   : 0, 6506516
    master (6.576M)   : 0, 6575703

    section Trace stats
    master (7.018M)   : 0, 7017522

    section Manual
    master (9.470M)   : 0, 9470281

    section Manual + Automatic
    This PR (6108) (5.925M)   : 0, 5924778
    master (6.183M)   : 0, 6182687

    section DD_TRACE_ENABLED=0
    master (9.027M)   : 0, 9027111
Loading 
gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6108) (10.363M)   : 0, 10362897
    master (10.268M)   : 0, 10268013
    benchmarks/2.9.0 (10.067M)   : 0, 10067315

    section Automatic
    This PR (6108) (6.717M)   : 0, 6717431
    master (6.723M)   : 0, 6723130
    benchmarks/2.9.0 (7.552M)   : 0, 7552193

    section Trace stats
    master (7.475M)   : 0, 7474976

    section Manual
    master (10.095M)   : 0, 10095486

    section Manual + Automatic
    This PR (6108) (6.153M)   : 0, 6153033
    master (6.275M)   : 0, 6275342

    section DD_TRACE_ENABLED=0
    master (9.507M)   : 0, 9506883
Loading

@bouwkast
bouwkast merged commit bd582eb into master Oct 2, 2024
@bouwkast
bouwkast deleted the conti/prefix-system-tests-env-vars branch October 2, 2024 17:59
@github-actions github-actions Bot added this to the vNext-v3 milestone Oct 2, 2024
agocs pushed a commit that referenced this pull request Oct 8, 2024
@wconti27
chore: prefix system tests env vars (#6108)
2b5c0dc 
## Summary of changes
Prefix system tests env vars to not conflict with local developer env
vars

## Reason for change

## Implementation details

## Test coverage

## Other details
<!-- Fixes #{issue} -->

<!-- ⚠️ Note: where possible, please obtain 2 approvals prior to
merging. Unless CODEOWNERS specifies otherwise, for external teams it is
typically best to have one review from a team member, and one review
from apm-dotnet. Trivial changes do not require 2 reviews. -->
@andrewlock andrewlock added the area:tests unit tests, integration tests label Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bouwkast bouwkast bouwkast approved these changes

Assignees

No one assigned

Labels

area:tests unit tests, integration tests

Projects

None yet

Milestone

3.4.0

Development

Successfully merging this pull request may close these issues.

3 participants

@wconti27 @andrewlock @bouwkast