I think one problem here (and in #41941 for Windows) is just that there's a limit to the number of concurrent tasks so even if they all get scheduled immediately, they still don't all actually run immediately. Other problems are that sometimes the runs fail then get rerun, which adds to the actual time it takes to "really" get started.

For example consider https://cirrus-ci.com/task/5406603418796032, which ran with compute credits. It spent 10 minutes on the first run which failed (no idea if we were charged for that run but the failure was during the clone so out of our control), then took another 13 minutes or so to get scheduled again. So that's 23 minutes before the test really got under way.

The Windows bots aren't running with compute credits, so the situation is correspondingly worse. For example consider https://cirrus-ci.com/task/6544684591087616 which took over 1h45m to start. Of the 11 shards on that PR, 9 took more than 15 minutes to get started (one had a similar cloning failure as above; the times for the long shards were roughly 15m, 15m, 30m, 45m, 50m, 1h10m, 1h35m, 1h45m, and 1h45m to start, the longest one to end took 2h15m to reach a result, of which 10m was setup and 20m testing).

On Linux it's less of an issue because the bots seem to all run in parallel; the 17 shards on Linux all started within about 20 minutes or so, without compute credits.

I'm not familiar with the costs and whether it would be practical to use compute credits for PRs for Windows (or whether it's really affordable for us to do it for Macs). We would like to dramatically increase our testing, the 41 or so shards we have today could easily be multiplied by two (for example we'd like to run lots of community tests, all the plugin's tests, run the analyzer on all the packages in pub, etc).

Anyway, this bug is tracking the fact that we have some mac tests turned off in some cases so that we don't run all 41 shards in all PRs.

macOS testing capacity is too low (macOS pre-commit tests disabled on cirrus) #41940

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions