I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

@fanyang-mono can you please point to the issue in dotnet/runtime related to this PR? We need that to track re-enabling the test once the external issue gets fixed.

Do you think this will disable tests corresponding for both JIT.jit64 and baservices.mono failures in https://dev.azure.com/dnceng/public/_build/results?buildId=1382481&view=results ?

external tracking issue: dotnet/xharness#663 (so it's easier to find this is connected)

fyi - @premun @greenEkatherine

Create a tracking issue in dotnet\runtime as a reminder to enable the tests again once external issue is fixed

Created #60128

@fanyang-mono is there actually an installation timeout happening still? I thought it's just the tests timing out for running too long now? There were failing installations because apps were left behind and the phones were full but no actually Android installation timeout, right?

@SamMonoRT, @fanyang-mono I have made a lot of improvements on how we time scope commands you run in Helix and how we collect telemetry. The only problem is that currently, Arcade hasn't been updated in runtime for a long time so we cannot get these things. Not sure who is responsible for that (runtime infra team?) but if we had the newest bits (from last week at least), we could actually measure each operation and see what is happening inside of the Xunit runner as it's hard to peek in.

The new updates will also give you the possibility to re-run the work item on a different Helix machine but overall I am afraid we are now rather hitting your tests running too long.

We are looking in the code and there's no installation timeouts. Furthermore it seems to be happening in the same set of apps such as baseservices.mono (for some it happens more, for some less). I also pulled Kusto logs and it is not tied to a specific machine - there's no set of machines that would cause this. I also see BCL running many times the number of work items you are and without a single timeout.

What's happening that we know about is that the Xunit runner calls XHarness to install the app, that finishes immediately. Then, after 3 hours, Xunit runner calls XHarness to run the app which starts running and then the Helix work item times out and the process is killed. We don't know yet what happens in those 3 hours.

I am not convinced this is actually infrastructural issue but there's no "external issue" that is being worked on that would fix this at the moment. Might be in the way Xunit runner calls things, might be somewhere in the Helix SDK but I think it needs active investigation. We will dig more with @fanyang-mono.

For the issue that @krwq is linking - that one can be resolved with the retry mechanism once we have the new Arcade and we make changes to request the retries.

@fanyang-mono @SamMonoRT I merged this as the queue got heavily back up with 3 hours work items today.
I think it's the right thing to disable the tests, I was just noting that the reason we are doing this is not what this PR claims.

We should bump Arcade in runtime (it has been already promoted and will soon be in main), open a PR to re-enable these and resolve the Xunit runner 3 hours timeout in the PR.