Skip to content

Experiment (PoC): Support VSTest-based --filter for MTP#3466

Closed
Youssef1313 wants to merge 2 commits into
xunit:mainfrom
Youssef1313:vstest-filter
Closed

Experiment (PoC): Support VSTest-based --filter for MTP#3466
Youssef1313 wants to merge 2 commits into
xunit:mainfrom
Youssef1313:vstest-filter

Conversation

@Youssef1313

Copy link
Copy Markdown
Contributor

This a PoC that's not yet tested. This PR adds --filter of VSTest so that people migrating from xunit 2 and VSTest to xunit.v3 and MTP can still run with --filter, which makes the migration much easier for many users.

@bradwilson I'm not sure if you're willing to accept --filter again in MTP, so I haven't put much efforts here yet.

@bradwilson

Copy link
Copy Markdown
Member

After looking at the amount of code involved, I'm reluctant to take on the maintenance burden that this would imply, for a feature that nobody has asked for. Switching your filters from VSTest to MTP is probably a relatively small amount of the effort involved in switching, and we wouldn't want to encourage people to use the outdated and aging filter syntax when superior options exist.

I believe that this feature probably belongs in core MTP rather than here, if this is a burden in moving from VSTest to MTP, since it would affect all testing frameworks and not just xUnit.net.

@Youssef1313

Copy link
Copy Markdown
Contributor Author

@bradwilson If we provide a source package with all the code in VSTestFiltering directory of this PR, would you be willing to accept this PR?

@Youssef1313

Copy link
Copy Markdown
Contributor Author

See also modelcontextprotocol/csharp-sdk#1011

@bradwilson

Copy link
Copy Markdown
Member

Who would write and maintain this library?

@bradwilson

Copy link
Copy Markdown
Member

Also, I am negatively motivated by "the AI bots are having a hard time" since I believe use of AI is at best unethical and and worst immoral.

@Youssef1313

Copy link
Copy Markdown
Contributor Author

Who would write and maintain this library?

I'm thinking of having this in VSTest repo, so that VSTest is the single source of truth.

This will also mean that we will be able to delete the relevant parts that we copied from vstest repo to the VSTestBridge implementation in testfx, which would be a win for us to avoid the duplication and double maintenance.

@bradwilson

Copy link
Copy Markdown
Member

@Youssef1313

Copy link
Copy Markdown
Contributor Author

@bradwilson That would be the logic in XunitFilters.cs in this PR. The core of it is this part which provides the values of properties.

			var filterExpression = new TestCaseFilterExpression(new FilterExpressionWrapper(vstestFilter));

			return filterExpression.MatchTestCase(propertyName =>
			{
				if (string.Equals(propertyName, "FullyQualifiedName", StringComparison.OrdinalIgnoreCase))
				{
					if (testCase.TestClassName is null || testCase.TestMethodName is null)
					{
						return null;
					}

					return $"{testCase.TestClassName}.{testCase.TestMethodName}"; // TODO: Should we add backtick and arity here?
				}
				else if (string.Equals(propertyName, "DisplayName", StringComparison.OrdinalIgnoreCase))
				{
					return testCase.TestCaseDisplayName;
				}

				_ = testCase.Traits.TryGetValue(propertyName, out var values);
				return values?.ToArray();
			});

@bradwilson

Copy link
Copy Markdown
Member

Having a singular implementation of the filter parsing (that is owned by Microsoft) certainly alleviates the maintenance concern.

@Youssef1313

Youssef1313 commented Jan 1, 2026

Copy link
Copy Markdown
Contributor Author

Great. I'll discuss with the team next week (or a bit after, depending on priorities).

@bradwilson

Copy link
Copy Markdown
Member

Any progress here?

@Youssef1313

Copy link
Copy Markdown
Contributor Author

Unfortunately, not yet.

@Youssef1313

Copy link
Copy Markdown
Contributor Author

Hi @bradwilson

We merged microsoft/vstest#15638

I updated the PR to use it.

<PackageReference Include="Microsoft.Win32.Registry" Version="$(Microsoft_Win32_Registry_Version)" />
<PackageReference Include="System.Security.AccessControl" Version="$(System_Security_AccessControl_Version)" />
<PackageReference Include="Mono.Cecil" Version="$(Mono_Cecil_Version)" PrivateAssets="all" />
<PackageReference Include="Microsoft.TestPlatform.Filter.Source" Version="18.7.0-preview-26213-05" PrivateAssets="all" />

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Youssef1313 Do you have a planned release date?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping @nohwnd

/// Gets the command-line arguments to pass to an xUnit.net v3 test assembly to perform
/// the filtering contained within this filter.
/// </summary>
// TODO: This is for xunit runner and not for MTP? Consider either supporting VSTest filter there or alternatively just throw when it's set as not supported?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an interesting situation.

Assuming we add a new switch to 4.0's in-process console runner (something like -vstestFilter), we would want first- and third-party runners be able to pass

We could add this to 4.0's in-process test runner without needing support here, since this is only used for third party or multi-assembly runners. We could pass in coreFrameworkVersion from Xunit3ArgumentFactory and only try to tack on the extra command line switch when we know the other side will support it (otherwise ignoring it).

We already have precedent for this with -culture:

-culture <option> : run tests under the given culture (v3 assemblies only)
                  : note: when running a v1/v2 assembly, the culture option will be ignored

We would just add similar wording here in our console runner. Third party runners would be responsible for knowing this feature is new to v3 4.0.0+, and if they're providing similar command line options, then it would be their responsibility to notify their users. Frankly, I don't think there are very many third party runners using our API beyond adapter-style runners (like our VSTest adapter), so command line options probably don't matter much at all to them.

@bradwilson

Copy link
Copy Markdown
Member

Integration here looks pretty simple on the surface. This seems like a reasonable POC. Once I know more about when we can count on this code coming from NuGet, then I'll look at moving this PR over to the rel/4.0.0 branch.

@bradwilson

Copy link
Copy Markdown
Member

@Youssef1313 @nohwnd I would like to get this integrated into 4.0, and I'm planning to do what I hope is a final prerelease this coming weekend, but I'm reluctant to take this dependency on a non-NuGet based source package (without the NuGet guarantee that the package will never be deleted).

@bradwilson

Copy link
Copy Markdown
Member

I have a version of this PR pushed to this branch: https://github.com/xunit/xunit/compare/rel/4.0.0...VSTestFilter?expand=1

This updated branch:

  • Targets rel/4.0.0 rather than main
  • Copies the source files from the NuGet package rather than take the dependency (until such time as the dependency is available on NuGet.org)
  • Adds -filterVSTest to the in-process runner command line
  • Adds missing unit tests

I've chosen not to add -filterVSTest to xunit.v3.runner.console at this time, due to the complications of trying to support it (or throw) for older projects. I suspect that -filterVSTest will not be frequently used, so I don't anticipate this to be an issue for most users, since it's expected that the primary use of this feature is via dotnet test or the MTP UX and --filter.

In-process console runner help:

dotnet test help:

@bradwilson

Copy link
Copy Markdown
Member

I'm just going to merge my branch as-is on the trust that the NuGet package will eventually show in the right place (or else the "implementation" of this feature will be frozen in time).

@bradwilson

Copy link
Copy Markdown
Member

Commit is here: 0765f56

Available in v3 4.0.0-pre.107 https://xunit.net/docs/using-ci-builds

Going to leave this PR open so I don't forget to periodically check for a release version of the NuGet package.

@bradwilson

Copy link
Copy Markdown
Member

@Youssef1313 @nohwnd Any hints on when an official package might show up on NuGet?

@Youssef1313

Youssef1313 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

@nohwnd Looks like VS 18.7 got released, but VSTest 18.7 isn't.

https://learn.microsoft.com/en-us/visualstudio/releases/2026/release-notes#18.7.1

Should VSTest 18.7 ship to nuget.org now? this release, I think, should contain the new package.

@nohwnd

nohwnd commented Jun 22, 2026

Copy link
Copy Markdown

18.7 is planned for tomorrow.

@bradwilson

Copy link
Copy Markdown
Member

Using the public NuGet package 1ef36bd

@bradwilson bradwilson closed this Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants