Improve CI: disable socket tests on Unix which may fail due to port being in-use#16548
Conversation
steveharter
added
area-System.Net.Sockets
test-run-core
|
cc @geoffkizer |
|
@steveharter would it help to make the offending tests exclusive on machine? Or is it overkill? |
|
Are you able to repro the same problematic behavior on Unix with a simple C app, for example, just using the POSIX APIs? I want to make sure the issue isn't actually in the .NET layer. |
|
@stephentoub I sent you a PM with the repro. |
|
test Innerloop Ubuntu14.04 Release Build and Test |
Not sure what you mean here. All Windows platforms work here, while all Unix\OSX platforms can fail here. |
|
I meant can we setup the tests to run in exclusive mode (at least on Linux), so that no other test executes on the VM in parallel? |
I see. Yes I think that would be overkill. I don't know of a good way to do that other than to create a new test assembly with [assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly, DisableTestParallelization = true, MaxParallelThreads = 1)]. Another approach is to run all such colliding tests in one test. Either way, I think these tests are flawed. Any other process (test or not) with one of the ports in use (that are not expected to be in use) can cause these types of tests to fail. Even in rare cases on Windows. |
|
@mellinoe is there a way to annotate a test to run exclusively without it being in separate test assembly? |
|
@karelz I don't think so. |
|
bummer :( thanks! |
6 participants
Several Socket tests bind to an open port (returned from the OS), and then connect to that same port but on a different IPV4 or IPV6 loopback address. On Windows that seems to work fine (that other loopback port is never in use), but on Unix (and OSX) port #'s are random (not incremental as in Windows) and it treats IPV4 and IPV6 as separate addresses thus the expected unused port will occasionally be in use causing that test, or related test to fail.
Most likely it is the 'mirror' test that is using the other protocol+port, meaning the test just before or after the current test that for example tests (IPV6, IPV4) while the original tests (IPV4, IPV6). I have repro'd the failures fairly consistently on Ubuntu by adding a 'for' loop to both tests so they execute 1,000 times or so. I have also written a small POSIX example that shows given a bound IPV6 loopback port, the OS can return an IPV4 address with the same port.
This PR just disables these tests for non-Windows cases. Note that these are also the 'failure' tests, meaning the happy path tests (IPV6, IPV6) or (IPV4, IPV4) are not changed here.
Addresses https://github.com/dotnet/corefx/issues/9017, https://github.com/dotnet/corefx/issues/13213
Also https://github.com/dotnet/corefx/issues/14519
cc @stephentoub , @CIPop, @davidsh