Skip to content

Testing: pytest should be running multiple nodes in one single process #19796

@lachaib

Description

@lachaib

Sorry if I missed an existing feature request with my searches I couldn't find one.

Hello,

When selecting multiple nodes to run tests, one would/could expect all of them to be run in one single process instantiation, for the following reasons:

  • performances: preparing some resources (data, loading modules on a big codebase...) may take some time and it is possible to have this time done once for all instead of once for each
  • reporting: one may want to get the coverage report generated by a selection of test nodes and coverage can be consolidated automatically with python-coverage instead of having to manually set coverage file option and then consolidate several files into one
  • isolation: it happens, despite efforts, that some tests don't behave the same when run together instead of one by one. To reproduce isolation issues that may be encountered later in CI, it'd be useful to be able to run in same conditions.

I've specified pytest in this issue although I think it may be valid for other runners, because I'm only using pytest and I know it's possible to specify several tests in command-line invocation.

Thinking about the case where people may expect parallel runs and consider current state as a feature, I think they should rather consider options like pytest-xdist instead.

I dug in the code, and, basically, it would mean that instead of having runTests do an await of delegated calls to runTest for each node, it may directly invoke pytest with all nodes in arguments.
Maybe even, if a parent and a child node are selected together only parent node could be passed to invocation, to avoid user mistakes leading to confusion.

In case this feature is accepted and needs external contribution, I may be able to code it

Thank you 🙂

Metadata

Metadata

Assignees

Labels

area-testingbugIssue identified by VS Code Team member as probable bug

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions