[don't merge] Try filtering out tests fails on net9#15450
Closed
nohwnd wants to merge 5 commits into
Closed
Conversation
dotnet.config is no longer suppported way of configuring mtp, replace it with global.json
Contributor
There was a problem hiding this comment.
Pull request overview
This PR attempts to avoid/diagnose unexpected failures on newer .NET (net9) by skipping a large set of acceptance/integration tests on .NET (Core) and by forcing those tests to run under the VSTest runner via a per-working-directory global.json.
Changes:
- Force
dotnet testintegration runs to use VSTest by writing/validating aglobal.jsonin the test working directory. - Add a new MSTest condition attribute and apply it broadly to skip many integration tests on .NET (Core) to avoid duplicated out-of-proc runs.
- Switch repo-level test runner configuration from
dotnet.configtoglobal.json.
Reviewed changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBase.cs | Writes/validates per-working-dir global.json to force VSTest runner |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestPlatformNugetPackageTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TestCaseFilterTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/TelemetryTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SkipIntegrationTestOnNetConditionAttribute.cs | Introduces new condition attribute used to filter test execution by runtime |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SelfContainedAppTests.cs | Skips integration test on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RunsettingsTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ResultsDirectoryTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/RecursiveResourcesLookupTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ProcessesInteractionTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PostProcessingTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PortableNugetPackageTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/PlatformTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/MultitargetingTestHostTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/LoggerTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ListExtensionsTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FrameworkTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/FilePatternParserTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionThreadApartmentStateTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/ExecutionTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/EventLogCollectorTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetTestMSBuildOutputTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetHostArchitectureVerifierTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DotnetArchitectureSwitchTests.Windows.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DiscoveryTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DisableAppdomainTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DifferentTestFrameworkSimpleTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DeprecateExtensionsPathWarningTests.cs | Adds a TODO comment in the test class header |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DebugAssertTests.cs | Skips integration test on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectorTests.Coverlet.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectorAttachmentsProcessorsFactoryTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/DataCollectionTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/CodeCoverageTests.cs | Skips integration tests on .NET (Core) via new condition attribute |
| test/Microsoft.TestPlatform.Acceptance.IntegrationTests/BlameDataCollectorTests.cs | Skips integration tests on .NET (Core) and adds a new test category |
| global.json | Adds repo-level test runner config using global.json |
| dotnet.config | Removes previous test runner config |
Comments suppressed due to low confidence (4)
test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBase.cs:1
workingDirectory!is dereferenced before the null-check, so callingInvokeDotnetTest(..., workingDirectory: null)will throw immediately (even though the followingifattempts to guard it). ComputeglobalJsonPathonly after verifyingworkingDirectoryis non-null, or early-return/throw with a clear message when it is null.
test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBase.cs:1- The JSON being written is invalid (trailing comma after
"vstest",and an extra comma before the closing brace). This will likely causedotnet testto ignore or error on the file, defeating the intent to enforce the runner. Write valid JSON (no trailing commas and correct brace structure).
test/Microsoft.TestPlatform.TestUtilities/IntegrationTestBase.cs:1 - Validating
global.jsonby substring match is brittle (formatting/whitespace, different property ordering, or equivalent JSON could fail validation). Prefer parsing JSON and checkingtest.runnersemantically, and consider usingEnvironment.NewLinefor message formatting consistency.
test/Microsoft.TestPlatform.Acceptance.IntegrationTests/SkipIntegrationTestOnNetConditionAttribute.cs:1 - The attribute name and internal field name are confusing relative to behavior:
ConditionMode.Include+_include = ...StartsWith(".NET Framework")means “run only on .NET Framework”, not “skip on net”. Consider renaming the attribute to reflect actual behavior (e.g.,RunOnlyOnNetFrameworkAttribute) and rename_includeto something explicit like_isNetFramework.
Member
Author
|
The failing test skips itself in Release mode, fixing in the linked PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I see local failures in unexpected place locally, trying if that will fail also in ci. if not something is very wrong.