Skip to content

Reduce parameterization matrix for most-executed acceptance tests #15470

Description

@nohwnd

Problem

The acceptance test suite has several tests with extremely broad parameterization matrices (runner × TFM × isolation combos), making them the most expensive tests by total wall-clock time. From a recent net48_x86 TRX analysis:

Runs Total sec Test Method
140 391.7 RunTestsFromMultipleMSTestAssemblies
71 291.6 RunMultipleTestAssemblies
40 RunTestsWithCustomTestHostLauncher…OutdatedITestHostLauncher
38 133.1 RunAllTestsWithMixedTFMs…
32 167.5 RunAllTests
32 142.8 RunMultipleMSTestAssemblies…Combinations2
32 129.1 DiscoverTestsUsingDiscoveryEventHandler1
32 129.1 RunTestsWithTestCaseFilter
32 115.3 DiscoverTestsUsingDiscoveryEventHandler2AndTelemetryOptedOut
28 134.5 RunTestsWithCustomTestHostLauncher…AttachesToDebugger

RunTestsFromMultipleMSTestAssemblies alone runs 140 times and consumes ~6.5 minutes of wall-clock time.

Suggestion

Review the parameterization for these high-frequency tests and determine whether the full matrix is necessary. Many of these tests likely validate framework-agnostic behavior and could run on a reduced set of runner/TFM combinations without losing meaningful coverage.

Possible approaches:

  • Reduce DataSource attributes to cover only boundary TFMs (e.g. oldest + newest)
  • Split into a "full matrix" category that runs only in CI, with a smaller matrix for local/PR runs
  • Deduplicate overlapping test scenarios (see also Deduplicate tests #15464)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

No fields configured for Task.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions