-
Notifications
You must be signed in to change notification settings - Fork 249
Description
The more I look at the scheduling problem, the more I think it should be solved using work stealing https://en.wikipedia.org/wiki/Work_stealing :
- Initially, all tests are split among the workers evenly.
- When some worker completes all of its tests, the scheduler asks the worker with max number of pending tests to give half of its pending tests back.
- After receiving the confirmation, the scheduler sends the returned tests to the worker that ran out of tests.
- Shutdown workers when no tests can be reallocated ("stolen").
Of course, there are some tricky synchronization details - but nothing impossible to implement. This algorithm shouldn't need any parameters or assumptions about test duration to perform optimally. And the code could even turn out to be simpler than the current LoadScheduling.
By "perform optimally" I mean:
- In the ideal case, where all test have the same duration:
ntests are split amongmworkers, each worker runsn/mconsecutive tests - and achieves the best reuse of fixtures possible (possible without additional information about the fixtures). - No workers are idle while there are pending tests.
The only issue is that I don't know when I'll have enough free time to write a custom scheduler from scratch.
I plan to work on this myself, but I'm not sure when I'll have time for it.
Also, I would appreciate feedback on the idea from the maintainers.