Based upon this PR/branch, OP text quoted here:

In the real world, test scopes aren't equal: Some have just a few tests which take little time to run while others can have a large amount of tests taking longer to finish.
With the current implementation it can happen that the scheduler assigns a scope with a large amount of tests very late in the run, often we see a single worker still running tests of a large scope while all other workers are idle because all other scopes have finished running.

This PR inserts scopes into the workqueue sorted by the amount of tests. The idea is to begin testing with the biggest scopes so that small scopes can essentially be bin packed late in the run so that all workers finish processing roughly at the same time.

I am aware that the number of tests is not the best indicator for test runtime but it is better than random scope assignment.
We have developed a more sophisticated version sorting with timings from previous test runs but unfortunately it requires external tooling.

I have one reservation about this change and backwards compatibility when we have test_a.py and test_b.py:

• in the current release, no test from test_b.py is run before any tests from test_a.py in any runner
• after this patch, tests from test_b.py might run before any tests from test_a.py in a runner (if test_b.py contains more tests)

I don't know what the general pytest(-plugin)'s view and pytest-xdist's view in particular is on backwards compatibility in the face of, essentially, flaky/broken tests.

Long term I wonder whether we would want to expose the ordering of this queue somehow: it would be cool when e.g. running in CircleCI (which can seed based on previous running times) to be able to repeat the same within the runners, especially as they could have many cores/parallel runners and so there can be inefficiencies.