This would just run on public helix nightly runs for now, test job here: https://dev.azure.com/dnceng/public/_build/results?buildId=1276727&view=results

It works! For now this just runs the tests in public Helix Nightly, and only on Linux (Fedora) - do we want to add them to PR builds?

There is one failing test, Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Tests.QuicStreamContextTests.BidirectionalStream_ServerWritesDataAndDisposes_ClientReadsData(dataLength: 1048576) (and dataLength: 5242880):

https://dev.azure.com/dnceng/public/_build/results?buildId=1276727&view=ms.vss-test-web.build-test-results-tab&runId=37760718&resultId=107727&paneView=debug

Assert.Equal() Failure
Expected: 0
Actual: 1

at System.IO.StreamExtensions.ReadUntilEndAsync(Stream stream, Int32 bufferLength, CancellationToken cancellationToken) in //src/Servers/Kestrel/shared/test/StreamExtensions.cs:line 87
at Microsoft.AspNetCore.Testing.TaskExtensions.TimeoutAfter[T](Task`1 task, TimeSpan timeout, String filePath, Int32 lineNumber) in //src/Shared/TaskExtensions.cs:line 90
at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Tests.QuicStreamContextTests.BidirectionalStream_ServerWritesDataAndDisposes_ClientReadsData(Int32 dataLength) in //src/Servers/Kestrel/Transport.Quic/test/QuicStreamContextTests.cs:line 104
at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Tests.QuicStreamContextTests.BidirectionalStream_ServerWritesDataAndDisposes_ClientReadsData(Int32 dataLength) in //src/Servers/Kestrel/Transport.Quic/test/QuicStreamContextTests.cs:line 109
at Microsoft.AspNetCore.Server.Kestrel.Transport.Quic.Tests.QuicStreamContextTests.BidirectionalStream_ServerWritesDataAndDisposes_ClientReadsData(Int32 dataLength) in /_/src/Servers/Kestrel/Transport.Quic/test/QuicStreamContextTests.cs:line 109
--- End of stack trace from previous location ---

Oh, wait, it is running in the helix matrix on PRs as well, but that's not a blocking check.

IMO preemptively mark ALL the tests in the Transport.Quic.Tests project and the Http3RequestTests file as quarantined.

System.Net.Quic is flaky, our bits are flaky, and tests are flaky. I don't want to block other people.

We'll keep an eye on the failure rates of the QUIC and HTTP/3 tests and address tests that fail often.

@Tratcher @adityamandaleeka

I don't think the helix nightly matrix runs the flaky tests, so to do that we'd have to include the fedora queue in the regular CI runs - is that right @HaoK?

it is running in the helix matrix on PRs as well, but that's not a blocking check.

?

?

aspnetcore-helix-matrix runs on PRs but doesn't block them on failure - only aspnetcore-ci is required.

Oh, never mind, it doesn't - it's just showing up here because the UI found my build of it against this branch, which I triggered manually. Though I also see a few that were automatically triggered for one of Hao's PRs today, which is also odd.

Yeah so I update the yml in some of my PRs that I want to run helix-matrix so I can see if it works everywhere, like the playwright PR

You can manually trigger a helix-matrix build in the azdo UI too, if you want to do a one off run on a PR

What I want:

• These tests to run often so we get an indication of which are flaky
• These tests not to block the build from successfully passing

Is there a way to do that?

We do have another pipeline aspnet-quarantined-tests that runs quarantined tests against main, so you can look at for example which runs on all the helix matrix queues: https://dev.azure.com/dnceng/public/_build/results?buildId=1276184&view=ms.vss-test-web.build-test-results-tab

The only issue you might have there is the volume of those runs is low since we no longer run them on timer, I think it only runs on commits to main. But that might be sufficient to get some idea if they are flaky or not

We also have the aspnetcore-quarantined-pr and that may be perfect for @JamesNK's wants

Ok, quic tests should now run only in aspnetcore-quarantined-pr

#35070

https://dev.azure.com/dnceng/public/_build/results?buildId=1278467&view=results - quic tests should not run here (helix matrix)

Looks like that worked. Quic tests now run only on fedora in aspnetcore-quarantined-pr (Tests: Helix)

@JamesNK @Tratcher any concerns with this?