-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Sorry if I missed an existing feature request with my searches I couldn't find one.
Hello,
When selecting multiple nodes to run tests, one would/could expect all of them to be run in one single process instantiation, for the following reasons:
- performances: preparing some resources (data, loading modules on a big codebase...) may take some time and it is possible to have this time done once for all instead of once for each
- reporting: one may want to get the coverage report generated by a selection of test nodes and coverage can be consolidated automatically with python-coverage instead of having to manually set coverage file option and then consolidate several files into one
- isolation: it happens, despite efforts, that some tests don't behave the same when run together instead of one by one. To reproduce isolation issues that may be encountered later in CI, it'd be useful to be able to run in same conditions.
I've specified pytest in this issue although I think it may be valid for other runners, because I'm only using pytest and I know it's possible to specify several tests in command-line invocation.
Thinking about the case where people may expect parallel runs and consider current state as a feature, I think they should rather consider options like pytest-xdist instead.
I dug in the code, and, basically, it would mean that instead of having runTests do an await of delegated calls to runTest for each node, it may directly invoke pytest with all nodes in arguments.
Maybe even, if a parent and a child node are selected together only parent node could be passed to invocation, to avoid user mistakes leading to confusion.
In case this feature is accepted and needs external contribution, I may be able to code it
Thank you 🙂