Experiment (PoC): Support VSTest-based --filter for MTP#3466
Experiment (PoC): Support VSTest-based --filter for MTP#3466Youssef1313 wants to merge 2 commits into
Conversation
84aa93c to
2d414d2
Compare
|
After looking at the amount of code involved, I'm reluctant to take on the maintenance burden that this would imply, for a feature that nobody has asked for. Switching your filters from VSTest to MTP is probably a relatively small amount of the effort involved in switching, and we wouldn't want to encourage people to use the outdated and aging filter syntax when superior options exist. I believe that this feature probably belongs in core MTP rather than here, if this is a burden in moving from VSTest to MTP, since it would affect all testing frameworks and not just xUnit.net. |
|
@bradwilson If we provide a source package with all the code in VSTestFiltering directory of this PR, would you be willing to accept this PR? |
|
See also modelcontextprotocol/csharp-sdk#1011 |
|
Who would write and maintain this library? |
|
Also, I am negatively motivated by "the AI bots are having a hard time" since I believe use of AI is at best unethical and and worst immoral. |
I'm thinking of having this in VSTest repo, so that VSTest is the single source of truth. This will also mean that we will be able to delete the relevant parts that we copied from vstest repo to the VSTestBridge implementation in testfx, which would be a win for us to avoid the duplication and double maintenance. |
|
And presumably test frameworks would need to implement something like we did for VSTest: https://github.com/xunit/visualstudio.xunit/blob/main/src/xunit.runner.visualstudio/Utility/TestCaseFilter.cs |
|
@bradwilson That would be the logic in var filterExpression = new TestCaseFilterExpression(new FilterExpressionWrapper(vstestFilter));
return filterExpression.MatchTestCase(propertyName =>
{
if (string.Equals(propertyName, "FullyQualifiedName", StringComparison.OrdinalIgnoreCase))
{
if (testCase.TestClassName is null || testCase.TestMethodName is null)
{
return null;
}
return $"{testCase.TestClassName}.{testCase.TestMethodName}"; // TODO: Should we add backtick and arity here?
}
else if (string.Equals(propertyName, "DisplayName", StringComparison.OrdinalIgnoreCase))
{
return testCase.TestCaseDisplayName;
}
_ = testCase.Traits.TryGetValue(propertyName, out var values);
return values?.ToArray();
}); |
|
Having a singular implementation of the filter parsing (that is owned by Microsoft) certainly alleviates the maintenance concern. |
|
Great. I'll discuss with the team next week (or a bit after, depending on priorities). |
|
Any progress here? |
|
Unfortunately, not yet. |
2d414d2 to
9e2820f
Compare
|
Hi @bradwilson We merged microsoft/vstest#15638 I updated the PR to use it. |
| <PackageReference Include="Microsoft.Win32.Registry" Version="$(Microsoft_Win32_Registry_Version)" /> | ||
| <PackageReference Include="System.Security.AccessControl" Version="$(System_Security_AccessControl_Version)" /> | ||
| <PackageReference Include="Mono.Cecil" Version="$(Mono_Cecil_Version)" PrivateAssets="all" /> | ||
| <PackageReference Include="Microsoft.TestPlatform.Filter.Source" Version="18.7.0-preview-26213-05" PrivateAssets="all" /> |
| /// Gets the command-line arguments to pass to an xUnit.net v3 test assembly to perform | ||
| /// the filtering contained within this filter. | ||
| /// </summary> | ||
| // TODO: This is for xunit runner and not for MTP? Consider either supporting VSTest filter there or alternatively just throw when it's set as not supported? |
There was a problem hiding this comment.
This is an interesting situation.
Assuming we add a new switch to 4.0's in-process console runner (something like -vstestFilter), we would want first- and third-party runners be able to pass
We could add this to 4.0's in-process test runner without needing support here, since this is only used for third party or multi-assembly runners. We could pass in coreFrameworkVersion from Xunit3ArgumentFactory and only try to tack on the extra command line switch when we know the other side will support it (otherwise ignoring it).
We already have precedent for this with -culture:
-culture <option> : run tests under the given culture (v3 assemblies only)
: note: when running a v1/v2 assembly, the culture option will be ignored
We would just add similar wording here in our console runner. Third party runners would be responsible for knowing this feature is new to v3 4.0.0+, and if they're providing similar command line options, then it would be their responsibility to notify their users. Frankly, I don't think there are very many third party runners using our API beyond adapter-style runners (like our VSTest adapter), so command line options probably don't matter much at all to them.
|
Integration here looks pretty simple on the surface. This seems like a reasonable POC. Once I know more about when we can count on this code coming from NuGet, then I'll look at moving this PR over to the |
|
@Youssef1313 @nohwnd I would like to get this integrated into 4.0, and I'm planning to do what I hope is a final prerelease this coming weekend, but I'm reluctant to take this dependency on a non-NuGet based source package (without the NuGet guarantee that the package will never be deleted). |
|
I have a version of this PR pushed to this branch: https://github.com/xunit/xunit/compare/rel/4.0.0...VSTestFilter?expand=1 This updated branch:
I've chosen not to add In-process console runner help:
|
|
I'm just going to merge my branch as-is on the trust that the NuGet package will eventually show in the right place (or else the "implementation" of this feature will be frozen in time). |
|
Commit is here: 0765f56 Available in v3 Going to leave this PR open so I don't forget to periodically check for a release version of the NuGet package. |
|
@Youssef1313 @nohwnd Any hints on when an official package might show up on NuGet? |
|
@nohwnd Looks like VS 18.7 got released, but VSTest 18.7 isn't. Should VSTest 18.7 ship to nuget.org now? this release, I think, should contain the new package. |
|
18.7 is planned for tomorrow. |
|
Using the public NuGet package 1ef36bd |


This a PoC that's not yet tested. This PR adds
--filterof VSTest so that people migrating from xunit 2 and VSTest to xunit.v3 and MTP can still run with--filter, which makes the migration much easier for many users.@bradwilson I'm not sure if you're willing to accept
--filteragain in MTP, so I haven't put much efforts here yet.