LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first by joekohlsdorf · Pull Request #632 · pytest-dev/pytest-xdist

joekohlsdorf · 2021-02-24T20:50:41Z

In the real world, test scopes aren't equal: Some have just a few tests which take little time to run while others can have a large amount of tests taking longer to finish.
With the current implementation it can happen that the scheduler assigns a scope with a large amount of tests very late in the run, often we see a single worker still running tests of a large scope while all other workers are idle because all other scopes have finished running.

This PR inserts scopes into the workqueue sorted by the amount of tests. The idea is to begin testing with the biggest scopes so that small scopes can essentially be bin packed late in the run so that all workers finish processing roughly at the same time.

I am aware that the number of tests is not the best indicator for test runtime but it is better than random scope assignment.
We have developed a more sophisticated version sorting with timings from previous test runs but unfortunately it requires external tooling.

… scope first

nicoddemus

Hi @joekohlsdorf,

Really sorry, seems we have missed this PR completely.

The idea sounds good to me, however in order to get this merge we also need a test that demonstrates this behavior (and fails without your patch of course). This ensures this works and avoids future regressions.

nicoddemus · 2021-11-28T13:40:31Z

Closing for now, feel free to reopen if you want to tackle this again. 👍

Bumps [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) from 3.4.0 to 3.5.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst">pytest-xdist's changelog</a>.</em></p> <blockquote> <h1>pytest-xdist 3.5.0 (2023-11-21)</h1> <h2>Features</h2> <ul> <li><code>[#632](pytest-dev/pytest-xdist#632) <https://github.com/pytest-dev/pytest-xdist/issues/632></code>_: <code>--dist=loadscope</code> now sorts scopes by number of tests to assign largest scopes early -- in many cases this should improve overall test session running time, as there is less chance of a large scope being left to be processed near the end of the session, leaving other workers idle.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/95b309e980796a261045d770f69c016ca741473d"><code>95b309e</code></a> Release 3.5.0</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/3fe877bc6d7211f31e55d2d561803f76ea95b935"><code>3fe877b</code></a> LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes ...</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/f36ea25cb53cb960037f4a8510816b81fa313035"><code>f36ea25</code></a> Create GitHub release after deploy (<a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/973">#973</a>)</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/d81f57508d064915fc16ace5dea4d43406db2ba0"><code>d81f575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/969">#969</a> from pytest-dev/pre-commit-ci-update-config</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/1d2616418c046d0e6f059b8f76e35a0627409616"><code>1d26164</code></a> [pre-commit.ci] pre-commit autoupdate</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/5f43575b42d6d4c0078de807880b8b0791b75823"><code>5f43575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/966">#966</a> from pytest-dev/release-3.4.0</li> <li>See full diff in <a href="https://github.com/pytest-dev/pytest-xdist/compare/v3.4.0...v3.5.0">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pytest-xdist&package-manager=pip&previous-version=3.4.0&new-version=3.5.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details>

) Updates the requirements on [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) to permit the latest version. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst">pytest-xdist's changelog</a>.</em></p> <blockquote> <h1>pytest-xdist 3.5.0 (2023-11-21)</h1> <h2>Features</h2> <ul> <li><code>[#632](pytest-dev/pytest-xdist#632) <https://github.com/pytest-dev/pytest-xdist/issues/632></code>_: <code>--dist=loadscope</code> now sorts scopes by number of tests to assign largest scopes early -- in many cases this should improve overall test session running time, as there is less chance of a large scope being left to be processed near the end of the session, leaving other workers idle.</li> </ul> <h1>pytest-xdist 3.4.0 (2023-11-11)</h1> <h2>Features</h2> <ul> <li> <p><code>[#963](pytest-dev/pytest-xdist#963) <https://github.com/pytest-dev/pytest-xdist/issues/963></code>_: Wait for workers to finish reporting when test run stops early.</p> <p>This makes sure that the results of in-progress tests are displayed. Previously these reports were being discarded, losing information about the test run.</p> </li> <li> <p><code>[#965](pytest-dev/pytest-xdist#965) <https://github.com/pytest-dev/pytest-xdist/issues/965></code>_: Added support for Python 3.12.</p> </li> </ul> <h1>pytest-xdist 3.3.1 (2023-05-19)</h1> <h2>Bug Fixes</h2> <ul> <li> <p><code>[#907](pytest-dev/pytest-xdist#907) <https://github.com/pytest-dev/pytest-xdist/issues/907></code>_: Avoid remote calls during startup as <code>execnet</code> by default does not ensure remote affinity with the main thread and might accidentally schedule the pytest worker into a non-main thread, which breaks numerous frameworks, for example <code>asyncio</code>, <code>anyio</code>, <code>PyQt/PySide</code>, etc.</p> <p>A more safe correction will require thread affinity in <code>execnet</code> (<code>pytest-dev/execnet#96 <https://github.com/pytest-dev/execnet/issues/96></code>__).</p> </li> </ul> <h1>pytest-xdist 3.3.0 (2023-05-12)</h1> <h2>Features</h2> <ul> <li><code>[#555](pytest-dev/pytest-xdist#555) <https://github.com/pytest-dev/pytest-xdist/issues/555></code>_: Improved progress output when collecting nodes to be less verbose.</li> </ul> <h1>pytest-xdist 3.2.1 (2023-03-12)</h1> <p>Bug Fixes</p>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/95b309e980796a261045d770f69c016ca741473d"><code>95b309e</code></a> Release 3.5.0</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/3fe877bc6d7211f31e55d2d561803f76ea95b935"><code>3fe877b</code></a> LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes ...</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/f36ea25cb53cb960037f4a8510816b81fa313035"><code>f36ea25</code></a> Create GitHub release after deploy (<a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/973">#973</a>)</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/d81f57508d064915fc16ace5dea4d43406db2ba0"><code>d81f575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/969">#969</a> from pytest-dev/pre-commit-ci-update-config</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/1d2616418c046d0e6f059b8f76e35a0627409616"><code>1d26164</code></a> [pre-commit.ci] pre-commit autoupdate</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/5f43575b42d6d4c0078de807880b8b0791b75823"><code>5f43575</code></a> Merge pull request <a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/966">#966</a> from pytest-dev/release-3.4.0</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/a6b56112f9b686aed7a354e0d50ecc26ad2d9dfe"><code>a6b5611</code></a> Update CHANGELOG</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/64f9388f4164e9f64d98a7580933e958ea4a09d5"><code>64f9388</code></a> Add support for Python 3.12</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/230ba6ad1057574c9f3d42a97f890788cd9ec6c3"><code>230ba6a</code></a> Properly wait for workers when test run terminates early (<a href="https://redirect.github.com/pytest-dev/pytest-xdist/issues/963">#963</a>)</li> <li><a href="https://github.com/pytest-dev/pytest-xdist/commit/93ca202b400eecc62919bcd9b0bab001e3f3f7ef"><code>93ca202</code></a> fix typo index -> instead</li> <li>Additional commits viewable in <a href="https://github.com/pytest-dev/pytest-xdist/compare/v2.1.0...v3.5.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

perf(scheduler): sort scopes by number of tests to start with biggest…

b131e41

… scope first

joekohlsdorf changed the title ~~feature: LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first~~ LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first Feb 24, 2021

nicoddemus requested changes Apr 27, 2021

View reviewed changes

nicoddemus mentioned this pull request Apr 27, 2021

Randomize load scope scheduler #536

Closed

nicoddemus closed this Nov 28, 2021

cryvate mentioned this pull request Apr 21, 2022

LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first #778

Merged

This was referenced Dec 12, 2023

Using pytest-xdist config: --dist=loadscope celery/pytest-celery#109

Merged

[Smoke Tests only] Using pytest-xdist config: --dist=loadscope celery/celery#8719

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first#632

LoadScope scheduler: Sort scopes by number of tests to assign biggest scopes first#632
joekohlsdorf wants to merge 1 commit intopytest-dev:masterfrom
cornershop:perf/sort_scopes_by_number_of_tests

joekohlsdorf commented Feb 24, 2021

Uh oh!

nicoddemus left a comment •

edited

Loading

Uh oh!

nicoddemus commented Nov 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

joekohlsdorf commented Feb 24, 2021

Uh oh!

nicoddemus left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicoddemus commented Nov 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nicoddemus left a comment •

edited

Loading