I'm not sure how to proceed here. @alexperovich , SignCheck only runs on framework. I was trying to create the SignCheck test project so that it would build / run on framework as well, but Helix seems to not like this and tries to default a lot of things to running on Core. Does that make sense? Is there any way to make this work? I've tried a couple of private builds with various tweaks but have been unable to find a way to make this work.

Helix by default runs dotnet xunit.console.dll <published test dll> or xunit.console.exe <test dll> based on the XUnitPublishTargetFramework and XUnitRuntimeTargetFramework. The former is "what TFM to dotnet publish the test project for" and the latter is "what TFM of xunit.console do I use". These can be overridden by TargetFramework and RuntimeTargetFramework metadata on the XUnitProject items. I believe you will need to set these metadata on the XUnitProject for the signcheck tests.

@alexperovich @ChadNedzlek

Trying to get the unit tests running for SignCheck was way more difficult than I had expected. Two reasons for the difficulty

• SignCheck is framework only, doesn't run on core
• SignCheck is x86

In Arcade, there didn't appear to be a straightforward way to say, "run this project on framework only using the xunit x86 console runner", primarily because it appears to me that all of the xunitproject items that are sent to a helixtargetqueue are expected to have the same characteristics.

I ended up creating an additional helix queue to run signcheck tests, and I had to update CreateXUnitWorkItems.cs to support an x86 identifier because trying to run the x86 SignCheck.Tests.dll using the x64 XUnit console runner just throw a "BadImageFormatException".

All of that is to say, that I'm not married to these changes (I just got stubborn about making things work), and it kind of changes how our Helix xunit-runner system works because it introduces the concept of an "XUnit architecture" (x86 vs x64). I was tempted to just omit these tests from running in Helix because it ended up being considerably more effort than it originally seemed. I'm completely ok with someone saying any of these things

• "Chris, that's trash, and we're not going to support x86 xunit, so omit them"
• "Chris, that's trash, we need to find a better solution, file an issue and disable the tests for now"
• "Chris, it's not a big deal, your solution is fine"
• "Chris, there's an easier way, we just didn't want to tell you before you got to understand the xunit runner better"
• "Chris, you don't actually get paid per written line of code, you need to talk to your agent"
• "Chris, stop typing"
• Other

If these changes are going to be merged, I have to break up this PR because the changes to CreateXUnitWorkItems.cs need to be available in the version of Arcade that the other changes consume.

You shouldn't need to break up the PR. Arcade sends to helix using the built version of itself. That being said, I don't like this much extra code to support a corner case of an unsupported runner. I think the better change is to use dotnet test inside the workitems. We already have dotnet cli on the machines, and dotnet test fully supports running on whatever architecture is needed to run the tests.

Do you have a pointer to a doc or an example of that. I got to this point because it appears that Arcade is explicitly using the method that I updated - https://github.com/dotnet/arcade/blob/master/tests/UnitTests.proj#L43. Maybe an example would help me see how Arcade's Helix testing and your comments align, because something doesn't quite mesh yet (in my head).

The support in helix is using xunit.console. What I was saying is I would rather we change that to use dotnet test to support this rather than continuing to use xunit.console.

Ok, that makes sense and aligns with my understanding. It feels to me like this is closer to...

"Chris, that's trash, we need to find a better solution, file an issue and disable the tests for now"

ie, switching to dotnet test would be outside the scope of what I'm working on. A tentative search, it appears you're referring to #4539, which isn't funded (at the moment).

If we don't want to update xunit console runner, then I'm reticent to merge anything here and potentially have a fractured testing experience between local dev box and CI systems. I guess I'll sit on this PR for the moment. :/

FYI @markwilkie, just as a data point

This actually isn't that issue. #4539 is referring to the run-tests-during-build support that is in the arcade sdk. It isn't referring to helix.

I see, I interpreted that issue as, let's use vs test everywhere and deprecate xunit runner, ie implying Helix. I can file another issue.

Closing this PR, it can be revisited after #6858 is resolved