testing: Use nextest for unit-tests by jschwe · Pull Request #39812 · servo/servo

jschwe · 2025-10-12T12:15:45Z

Nextest is a powerful test runner, with many advantages over cargo test. Among others it will run each test in a separate process, provide a summary of the completed test execution, supports output formats like JUnit, and can handle flaky test by retrying. This PR does not use most of these advanced features yet though, that will be left to future PRs.

The change also uncovered racy tests in net which rely on someone initializing the ASYNC_RUNTIME before the test in questions is run. By explicitly accessing the lazy static in a common setup routine, we make sure it is initialized.

CLI differences of cargo test vs cargo nextest:

The --profile option in nextest is not the cargo profile, but refers to nextest configuration profiles.
--nocapture is a direct argument to nextest, not to the testharness, so we can remove the preceding -- seperator.

Testing: Tested by running unit-tests in CI
Fixes: #39797

mukilan

Do you mind adding cargo-nextest to shell.nix as well, like we have for cargo-deny? I've tested on my Mac where I use nix and it seems to work with the version we get from nixpkgs which is 0.9.105.

components/servo/tests/common/mod.rs

python/servo/platform/base.py

mukilan · 2025-10-14T13:17:45Z

python/servo/testing_commands.py

            pass  # there is no argument for debug
        else:
-            args += ["--profile", build_type.profile]
+            args += ["--cargo-profile", build_type.profile]


What is rationale for this change?

The --profile parameter for nextest is unfortunately not the same as a the --profile parameter for cargo. The nextest profile is for essentially choosing a test profile / configuration (e.g. CI). So when switching from cargo test to cargo nextest one needs to use --cargo-profile instead of --profile

I updated the PR description to mention these changes, since the commits will be squashed and the original commit messages lost.

python/servo/platform/base.py

Nextest is a powerful test runner, with many advantages of cargo test. Among others it will run each test in a separate process, provide a summary of the completed test execution, supports output formats like JUnit, and can handle flaky test by retrying. Signed-off-by: Jonathan Schwender <[email protected]>

nextest takes the argument directly, not the test harness (difference to cargo test). We also don't want to fail fast and run all tests. Signed-off-by: Jonathan Schwender <[email protected]>

The --profile option in nextest is not the cargo profile, but refers to nextest configuration profiles. Signed-off-by: Jonathan Schwender <[email protected]>

Signed-off-by: Jonathan Schwender <[email protected]>

mukilan

Thanks!

sagudev

❤️

Signed-off-by: Jonathan Schwender <[email protected]>

.github/workflows/linux.yml

Signed-off-by: Jonathan Schwender <[email protected]>

Follow-up to #39812, using nextests builtin retry feature to rerun flaky unit-tests. This also adds a per-test timeout, replacing the global timeout set for the retry action. We could also add a global timeout for nextest tests, but per-test timeouts should be sufficient and noticably speedup CI when individual unit tests get stuck. Testing: [mach try](https://github.com/servo/servo/actions/runs/18519094535) running the unit-tests with retries in CI. --------- Signed-off-by: Jonathan Schwender <[email protected]>

The changes in servo#39812 seem to have completely disabled the API tests, so this PR gets them running again by removing the custom test harness (apart from the multiprocess test). The custom test harness is typically not necessary now that all tests are run in their own process. In addition, the behavior of the `WebView::page_zoom()` is fixed when set via the `viewport` meta tag. This functionality was broken, but the unit test wasn't catching it. There is still one failing test, but that is tracked via servo#40129. Signed-off-by: Martin Robinson <[email protected]>

) The changes in #39812 seem to have completely disabled the API tests, so this PR gets them running again by removing the custom test harness (apart from the multiprocess test). The custom test harness is typically not necessary now that all tests are run in their own process. In addition, the behavior of the `WebView::page_zoom()` is fixed when set via the `viewport` meta tag. This functionality was broken, but the unit test wasn't catching it. There is still one failing test, but that is tracked via #40129. Testing: This change fixes the tests. Signed-off-by: Martin Robinson <[email protected]>

servo-highfive added the S-awaiting-review There is new code that needs to be reviewed. label Oct 12, 2025

This was referenced Oct 12, 2025

ci: Use nextest --retries in CI #39817

Merged

Unit tests get stuck in self-hosted MacOS runners #39839

Closed

mukilan reviewed Oct 14, 2025

View reviewed changes

jschwe added 6 commits October 14, 2025 16:12

Fix --nocapture and add --no-fail-fast

5874369

nextest takes the argument directly, not the test harness (difference to cargo test). We also don't want to fail fast and run all tests. Signed-off-by: Jonathan Schwender <[email protected]>

Use --cargo-profile instead of --profile

b776945

The --profile option in nextest is not the cargo profile, but refers to nextest configuration profiles. Signed-off-by: Jonathan Schwender <[email protected]>

add cargo-nextest to shell.nix

305e969

Signed-off-by: Jonathan Schwender <[email protected]>

format macro

c90a72d

Signed-off-by: Jonathan Schwender <[email protected]>

apply review suggestions

a14268c

Signed-off-by: Jonathan Schwender <[email protected]>

jschwe force-pushed the jschwender/nextest branch from 3a82adb to a14268c Compare October 14, 2025 14:19

mukilan approved these changes Oct 14, 2025

View reviewed changes

servo-highfive removed the S-awaiting-review There is new code that needs to be reviewed. label Oct 14, 2025

sagudev approved these changes Oct 14, 2025

View reviewed changes

jschwe enabled auto-merge October 14, 2025 14:45

jschwe added this pull request to the merge queue Oct 14, 2025

servo-highfive added the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Oct 14, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 14, 2025

servo-highfive added S-tests-failed The changes caused existing tests to fail. and removed S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. labels Oct 14, 2025

install nextest in ci

bcf197b

Signed-off-by: Jonathan Schwender <[email protected]>

servo-highfive added S-awaiting-review There is new code that needs to be reviewed. and removed S-tests-failed The changes caused existing tests to fail. labels Oct 14, 2025

sagudev reviewed Oct 14, 2025

View reviewed changes

.github/workflows/linux.yml Show resolved Hide resolved

jschwe added this pull request to the merge queue Oct 14, 2025

servo-highfive added the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Oct 14, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Oct 14, 2025

servo-highfive added S-tests-failed The changes caused existing tests to fail. and removed S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. labels Oct 14, 2025

install cargo nextest from source on windows

cc63f3d

Signed-off-by: Jonathan Schwender <[email protected]>

servo-highfive removed the S-tests-failed The changes caused existing tests to fail. label Oct 14, 2025

jschwe enabled auto-merge October 14, 2025 19:14

jschwe added this pull request to the merge queue Oct 14, 2025

servo-highfive added the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Oct 14, 2025

Merged via the queue into servo:main with commit 4b57867 Oct 14, 2025
28 checks passed

jschwe deleted the jschwender/nextest branch October 14, 2025 20:31

servo-highfive removed the S-awaiting-merge The PR is in the process of compiling and running tests on the automated CI. label Oct 14, 2025

sagudev mentioned this pull request Oct 15, 2025

Fix coverage job on self-hosted runners by installing nextest. #39895

Merged

delan mentioned this pull request Oct 16, 2025

Add additional preinstalled tools servo/ci-runners#54

Open

3 tasks

This was referenced Oct 24, 2025

libservo: Merge input method activation into the EmbedderControl API #40014

Merged

WebView test_theme_change is failing #40129

Closed

mrobinson mentioned this pull request Oct 24, 2025

libservo: Stop using the custom test harness and run_api_tests #40131

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

testing: Use nextest for unit-tests#39812

testing: Use nextest for unit-tests#39812
jschwe merged 8 commits intoservo:mainfrom
jschwe:jschwender/nextest

jschwe commented Oct 12, 2025 •

edited

Loading

Uh oh!

mukilan left a comment

Uh oh!

Uh oh!

Uh oh!

mukilan Oct 14, 2025

Uh oh!

jschwe Oct 14, 2025

Uh oh!

jschwe Oct 14, 2025

Uh oh!

Uh oh!

mukilan left a comment

Uh oh!

sagudev left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

jschwe commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mukilan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mukilan Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jschwe Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

jschwe Oct 14, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mukilan left a comment

Choose a reason for hiding this comment

Uh oh!

sagudev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jschwe commented Oct 12, 2025 •

edited

Loading