perf(browser): improve browser parallelisation by sheremet-va · Pull Request #7665 · vitest-dev/vitest

sheremet-va · 2025-03-12T14:48:49Z

Description

Fixes #7616
Fixes #6834

This made tests run 2 as fast as before in PW provider for me

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
Ideally, include a test that fails without this PR but passes with it.
Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

Run the tests with pnpm test:ci.

Documentation

If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

netlify · 2025-03-12T15:12:53Z

✅ Deploy Preview for vitest-dev ready!

Name	Link
🔨 Latest commit	`c58a06e`
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/67eab87debd281000854d2b0
😎 Deploy Preview	https://deploy-preview-7665--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

pkg-pr-new · 2025-03-13T15:50:27Z

@vitest/browser

npm i https://pkg.pr.new/@vitest/browser@7665

@vitest/coverage-istanbul

npm i https://pkg.pr.new/@vitest/coverage-istanbul@7665

@vitest/coverage-v8

npm i https://pkg.pr.new/@vitest/coverage-v8@7665

@vitest/expect

npm i https://pkg.pr.new/@vitest/expect@7665

@vitest/mocker

npm i https://pkg.pr.new/@vitest/mocker@7665

@vitest/pretty-format

npm i https://pkg.pr.new/@vitest/pretty-format@7665

@vitest/runner

npm i https://pkg.pr.new/@vitest/runner@7665

@vitest/spy

npm i https://pkg.pr.new/@vitest/spy@7665

@vitest/snapshot

npm i https://pkg.pr.new/@vitest/snapshot@7665

@vitest/ui

npm i https://pkg.pr.new/@vitest/ui@7665

@vitest/utils

npm i https://pkg.pr.new/@vitest/utils@7665

vite-node

npm i https://pkg.pr.new/vite-node@7665

vitest

npm i https://pkg.pr.new/vitest@7665

@vitest/web-worker

npm i https://pkg.pr.new/@vitest/web-worker@7665

@vitest/ws-client

npm i https://pkg.pr.new/@vitest/ws-client@7665

commit: c58a06e

sheremet-va · 2025-03-13T15:52:08Z

@nstepien can you check if this improves things for you? watch mode, no-isolation and debugging don't work properly now, but I think just running with --run should work

nstepien · 2025-03-13T16:12:48Z

With our unchanged config:

npx vitest run

 RUN  v3.0.8 D:/repo

4:04:53 PM [vite] (client) Re-optimizing dependencies because lockfile has changed
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Errors ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

Vitest caught 1 unhandled error during the test run.
This might cause false positive tests. Resolve unhandled errors to make sure your tests are not affected.

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Unhandled Error ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯
Error: [vitest-api]: Timeout calling "createTesters"
 ❯ Object.onTimeoutError node_modules/@vitest/browser/dist/index.js:3010:17
 ❯ Timeout._onTimeout node_modules/@vitest/browser/dist/index.js:2715:41
 ❯ listOnTimeout node:internal/timers:614:17
 ❯ process.processTimers node:internal/timers:549:7

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯


 Test Files   (232)
      Tests  53 passed (65)
     Errors  1 error
   Start at  16:04:53
   Duration  74.49s (transform 0ms, setup 1.57s, collect 2.46s, tests 0ms, environment 0ms, prepare 1.07s)

At first I though it was breaking the default reporter, but it's actually only running one test suite at a time.
Not sure if I need to configure anything differently, but this doesn't work well.

nstepien · 2025-03-13T16:18:01Z

Another run in a subset of our codebase:

npx vitest run src/camp

 RUN  v3.0.8 D:/repo

 ✓  chromium  src/....test.tsx (13 tests) 13214ms
 ✓  chromium  src/....test.tsx (6 tests) 8096ms
 ✓  chromium  src/....test.tsx (11 tests) 7908ms
 ✓  chromium  src/....test.tsx (3 tests) 2447ms
 ✓  chromium  src/....test.tsx (3 tests) 1469ms
                                                                                              
 Test Files  5 passed (5)                                                                     
      Tests  36 passed (36)                                                                   
   Start at  16:16:19                                                                         
   Duration  46.19s (transform 0ms, setup 2.45s, collect 1.50s, tests 33.13s, environment 0ms, prepare 528ms)

it runs to completion, but it still only runs one test suite at a time.

sheremet-va · 2025-03-13T16:29:35Z

it runs to completion, but it still only runs one test suite at a time.

This was a mistake on my part. Last minute if broke the parallelisation because of isolate: false

At first I though it was breaking the default reporter, but it's actually only running one test suite at a time.

Does it print any tests running? It seems like there is a timeout for createTesters - there shouldn't be one

nstepien · 2025-03-13T16:32:36Z

Does it print any tests running?

It does, seems like the default reporter behaves as expected.

nstepien · 2025-03-13T17:05:24Z

The latest build works better, but I still get the Timeout calling "createTesters" error before I can confirm whether parallelization has been fixed.

nstepien · 2025-03-14T17:14:03Z

The latest builds work great now 👍

sheremet-va · 2025-03-14T17:42:21Z

The latest builds work great now 👍

Is it any faster than before?

nstepien · 2025-03-14T17:45:09Z

Absolutely, yes

sheremet-va · 2025-03-14T17:47:52Z

Absolutely, yes

Can you share the numbers?

nstepien · 2025-03-14T17:54:05Z

In our CI, before:

Duration  800.08s (transform 0ms, setup 679.36s, collect 877.16s, tests 1486.22s, environment 0ms, prepare 207.40s)

After, with maxWorkers: 8 because tests somehow become flaky with increased parallelization, we need to fix our tests:

Duration  360.87s (transform 0ms, setup 364.73s, collect 483.01s, tests 1519.42s, environment 0ms, prepare 64.00s)

Once we fix our flaky tests it'll be faster still.

…-parallelisation

sheremet-va · 2025-03-26T10:10:02Z

With this change, firefox now fails consistently on Windows 😞 I am also getting weird time outs in different tests

I think we should wait for #7377 to be fixed first before releasing this

…-parallelisation

v3.1.2 🐞 Bug Fixes Add global chai variable in vitest/globals (fix: #7474) - by @Jay-Karia in vitest-dev/vitest#7771 and vitest-dev/vitest#7474 (d9297) Prevent modifying test.exclude when same object passed in coverage.exclude - by @AriPerkkio in vitest-dev/vitest#7774 (c3751) Fix already hoisted mock - by @hi-ogawa in vitest-dev/vitest#7815 (773b1) Fix test.scoped inheritance - by @hi-ogawa in vitest-dev/vitest#7814 (db6c3) Remove pointer-events-none after resizing the left panel - by @alexprudhomme in vitest-dev/vitest#7811 (a7e77) Default to run mode when stdin is not a TTY - by @kentonv, @hi-ogawa and @sheremet-va in vitest-dev/vitest#7673 (6358f) Use happy-dom/jsdom types for envionmentOptions - by @hi-ogawa in vitest-dev/vitest#7795 (67430) browser: Fix transform error before browser server initialization - by @hi-ogawa in vitest-dev/vitest#7783 (5f762) Fix mocking from outside of root - by @hi-ogawa in vitest-dev/vitest#7789 (03f55) Scale iframe for non ui case - by @hi-ogawa in vitest-dev/vitest#6512 (c3374) coverage: await profiler calls - by @AriPerkkio in vitest-dev/vitest#7763 (795a6) Expose profiling timers - by @AriPerkkio in vitest-dev/vitest#7820 (5652b) deps: Update all non-major dependencies - in vitest-dev/vitest#7765 (7c3df) Update all non-major dependencies - in vitest-dev/vitest#7831 (15701) runner: Correctly call test hooks and teardown functions - by @sheremet-va in vitest-dev/vitest#7775 (3c00c) Show stacktrace on test timeout error - by @hi-ogawa in vitest-dev/vitest#7799 (df33b) ui: Load panel sizes from storage on initial load - by @userquin in vitest-dev/vitest#7265 (6555d) vite-node: Named export should overwrite export all - by @hi-ogawa in vitest-dev/vitest#7846 (5ba0d) Add ERR_MODULE_NOT_FOUND code error if module cannot be loaded - by @sheremet-va in vitest-dev/vitest#7776 (f9eac) 🏎 Performance browser: Improve browser parallelisation - by @sheremet-va in vitest-dev/vitest#7665 (816a5)

mleimer · 2025-07-03T15:21:06Z

I understand that this fix shall run multiple test files in parallel in browser-mode. However, I am unable to achieve that in a non-headless mode, i.e. only one test file is running at a time.

Repository
https://github.com/mleimer/vitest-browser-parallelism

Do I misunderstand anything or what am I missing out?

When setting browser.headless: true, the test files indeed run in parallel. Is there anyway to achieve the same without using headless?

sheremet-va · 2025-07-03T16:14:37Z

I understand that this fix shall run multiple test files in parallel in browser-mode. However, I am unable to achieve that in a non-headless mode, i.e. only one test file is running at a time.

Repository https://github.com/mleimer/vitest-browser-parallelism

Do I misunderstand anything or what am I missing out?

When setting browser.headless: true, the test files indeed run in parallel. Is there anyway to achieve the same without using headless?

Tests run in parallel only in headless mode.

sheremet-va added 2 commits March 12, 2025 15:47

perf(browser): improve browser parallelisation

32d5726

chore: undo lockfile change

1a47659

sheremet-va added 3 commits March 13, 2025 15:32

refactor: a lot of stuff

bd16578

chore: cleanup

8253d5d

chore: restrict isolated parallelism for now

8c64611

sheremet-va added the cr-tracked label Mar 13, 2025

chore: cleanup

18fe445

sheremet-va added 5 commits March 13, 2025 18:25

chore: cleanup

6a2ae8d

fix: support dynamic context

3e4fe4b

fix: remove timeout from createTesters

6e8c1db

chore: cleanup

16fc7ac

fix: run non-isolated tests correctly

1b8b1fb

sheremet-va added 2 commits March 14, 2025 19:18

fix: correctly fail the test run if html threw an error

bbf0087

chore: chore

3b2d902

sheremet-va added the p3-significant High priority enhancement (priority) label Mar 17, 2025

sheremet-va added 2 commits March 24, 2025 17:08

Merge branch 'main' of github.com:vitest-dev/vitest into perf/browser…

3d5bc68

…-parallelisation

chore: fix snapshot

e67a7eb

sheremet-va added 6 commits March 24, 2025 19:08

fix: correctly run tests

3275269

feat: update the error message when the page crushes

05ae561

chore: support array in toReportPassedTest

d7559b7

Merge branch 'main' of github.com:vitest-dev/vitest into perf/browser…

af78258

…-parallelisation

refactor: cleanup pool

57c78ae

chore: debug logs in setup file

4f13278

sheremet-va added the upstream label Mar 26, 2025

Merge branch 'main' into perf/browser-parallelisation

4a77297

sheremet-va mentioned this pull request Mar 31, 2025

perf(browser): improve the parallelisation logic #6856

Closed

6 tasks

sheremet-va added 2 commits March 31, 2025 17:44

Merge branch 'main' of github.com:vitest-dev/vitest into perf/browser…

a31b738

…-parallelisation

chore: cleanup

c58a06e

sheremet-va marked this pull request as ready for review April 17, 2025 11:22

sheremet-va merged commit 816a5c5 into vitest-dev:main Apr 17, 2025
12 of 14 checks passed

sheremet-va deleted the perf/browser-parallelisation branch April 17, 2025 11:22

This was referenced Apr 17, 2025

Reject a promise after a test will make test end earlier in browser mode. #7079

Closed

window.location.pathname contains test metadata when Browser mode is enabled #7424

Closed

renovate bot mentioned this pull request Apr 24, 2025

chore(deps): update pnpm to v10.10.0 elonehoo/elonehoo.me#123

Merged

1 task

nstepien mentioned this pull request Apr 24, 2025

Inexplicable and flaky userEvent action timeouts #7871

Open

6 tasks

This was referenced Apr 24, 2025

fix(deps): update all non-major dependencies groupultra/telegram-search#125

Merged

fix(deps): update all non-major dependencies TomPlum/sleep#57

Open

This was referenced May 26, 2025

chore(deps): update storybook marigold-ui/marigold#4626

Merged

chore(deps): update all non-major dependencies elonehoo/vue-mini-sweeper#186

Open

chore(deps): update all non-major dependencies - autoclosed sxzz/magic-string-ast#56

Closed

This was referenced Jun 3, 2025

chore(deps): update all non-major dependencies - autoclosed vuejs/babel-plugin-jsx#757

Closed

chore(deps): update pnpm to v10.13.1 unjs/jiti#382

Closed

renovate bot mentioned this pull request Aug 12, 2025

fix(deps): update all dependencies un-ts/cf-cleaner#64

Open

1 task

Uh oh!

Comments

Conversation

sheremet-va commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

Tests

Documentation

Changesets

Uh oh!

netlify bot commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vitest-dev ready!

Uh oh!

pkg-pr-new bot commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sheremet-va commented Mar 13, 2025

Uh oh!

nstepien commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nstepien commented Mar 13, 2025

Uh oh!

sheremet-va commented Mar 13, 2025

Uh oh!

nstepien commented Mar 13, 2025

Uh oh!

nstepien commented Mar 13, 2025

Uh oh!

nstepien commented Mar 14, 2025

Uh oh!

sheremet-va commented Mar 14, 2025

Uh oh!

nstepien commented Mar 14, 2025

Uh oh!

sheremet-va commented Mar 14, 2025

Uh oh!

nstepien commented Mar 14, 2025

Uh oh!

sheremet-va commented Mar 26, 2025

Uh oh!

Uh oh!

mleimer commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sheremet-va commented Jul 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sheremet-va commented Mar 12, 2025 •

edited

Loading

netlify bot commented Mar 12, 2025 •

edited

Loading

pkg-pr-new bot commented Mar 13, 2025 •

edited

Loading

nstepien commented Mar 13, 2025 •

edited

Loading

mleimer commented Jul 3, 2025 •

edited

Loading