also run quarantined tests?

@HaoK That part is important. Do you have a suggestion on how we do that? Do we need some new functionality within the build system to support that, or is there an existing flag or similar we could pass?

Update: see below.

Actually, I have a better proposal for how we handle the quarantined E2E tests:

1. We stop running the components E2E test as part of the main build pipeline
   • ... but in some way that doesn't also stop them from running in the new pipeline that @HaoK has added in this PR
2. We unquarantine all the components E2E tests unconditionally
3. For any of them that really fail in the new pipeline, we investigate and either fix or requarantine, depending on whether we think there's an easy fix

This is because the existing set of quarantines for E2E test are just random noise. The quarantinings in most cases don't reflect any reality about reliability; they are just a record of prior random failures due to unknown causes that might be infrastructural. It's likely that some of the tests do have genuine problems, but we have no reason to think it's all or even most of the quarantined ones.

What do you think, @HaoK? And how can we stop the components E2E tests from running in the main build pipeline without also stopping them running in your new pipeline? Once we solve that I think we could merge this PR 😄

Yeah that sounds reasonable to me, my guess is that if you enable all of the tests the failure rate will be very high but that's a good idea to start from no quarantine.

I think the easiest way to do what you want is to add a new property to the Components E2ETest csproj like this:

    <SkipTests Condition="'$(RunComponentTests)' != 'true'">true</SkipTests>

Similar to how template tests are selectively run as well: https://github.com/dotnet/aspnetcore/blob/main/src/ProjectTemplates/test/ProjectTemplates.Tests.csproj#L12

Then in the yaml for ci you just need to set it to false: add a new /p:RunComponentTests=false

https://github.com/dotnet/aspnetcore/blob/main/.azure/pipelines/ci.yml#L656

And we'd also add a /p:RunComponentTests=true to the components pipeline in this PR, and you'd also probably want to turn off any other tests that run in the windows test group (signalr? I'm not sure what else this runs today, but I know there's a few other remaining tests in there)

You don't actually need to unquarantine tests for this new pipeline either, you can just specify: /p:RunQuarantinedTests=true which will run them all

@HaoK Hope this is OK - I've added a further change here intended to complete what we need for now (disabling the e2e tests in the main build). It uses the technique you recommended.

Looks good to me, altho there's one further thing to consider since there also seems to be build reliability with the E2E test app, which is to skip building the project in the main ci.yml as well?

@HaoK I didn't realise there were any issues building the E2ETests project. That sounds serious and we would want to know if that goes wrong. I don't think we should be suppressing that kind of problem (e.g., maybe it reflects some actual SDK issue).

Also once you have this pipeline as follow-up, might be worth looking into how to inclue the blazor templates + new playwright e2e as part of this pipeline which are currently in quarantine on the helix legs

I didn't realise there were any issues building the E2ETests project. That sounds serious and we would want to know if that goes wrong. I don't think we should be suppressing that kind of problem (e.g., maybe it reflects some actual SDK issue).

Yeah its usually some variation of something in the basictest app being filed locked, sometimes when compression, sometimes during other build steps. We've had that issue for a long time in some form or another, I saw it on a few builds recently (both locally) and on CI

@HaoK Looking at the list of checks in the GitHub UI now, I see it no longer includes anything about this new pipeline. Have I broken it? I did rename the file from components.yml to components-e2e-tests.yml. Would that have somehow stopped it from running?

Yeah its usually some variation of something in the basictest app being filed locked, sometimes when compression, sometimes during other build steps. We've had that issue for a long time in some form or another, I saw it on a few builds recently (both locally) and on CI

That sounds like an issue we need to debug, not suppress.

Oh wait, I see - it's in the AzDO config. I'll try to update it.

Done - it's running now.

Looks like its green, so do you want me to merge this or are you going to merge this?

Merged 👍

Thanks for sorting out the main parts of this Hao!

Note this pipeline runs more than just components tests e.g. the overall ASP.NET Core functional tests. Was that intended❔

I pointed that out as well, but I think for the purposes of a tripwire for components, its probably harmless as is, since its unlikely that the other tests are going to be what causes this check to be red