Skip to content

Commit 88202ad

Browse files
authored
New flag --shard-by-suite
Running all setupAll hooks is expensive when what we are trying to do is to minimize the time spent for each shard. This allows choosing a less fine-grained strategy of sharding by individual test files. Fixes dart-lang#2614
1 parent c80d53d commit 88202ad

10 files changed

Lines changed: 656 additions & 105 deletions

File tree

‎pkgs/test/CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
all tests with OS `'windows'` would previously still run browser tests on
5151
windows, but will now skip all tests including browser tests.
5252
* Use a DevTools URL instead of a defunct observatory URL.
53+
* Add flag `--shard-by-suite` to control sharding strategy.
5354
* Disable throttling in chrome launch arguments.
5455
* Allow package_config `3.x.x`.
5556
* Require `analyzer: '>=13.0.0 <15.0.0'`

‎pkgs/test/README.md‎

Lines changed: 25 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -204,12 +204,31 @@ dart test --total-shards 3 --shard-index 0 path/to/test.dart
204204
dart test --total-shards 3 --shard-index 1 path/to/test.dart
205205
dart test --total-shards 3 --shard-index 2 path/to/test.dart
206206
```
207-
Sharding: This refers to the process of splitting up a large test suite into
208-
smaller subsets (called shards) that can be run independently. Sharding is
209-
particularly useful for distributed testing, where multiple machines are used
210-
to run tests simultaneously. By dividing the test suite into smaller subsets,
211-
you can run tests in parallel across multiple machines, which can significantly
212-
reduce the overall testing time.
207+
208+
By default, sharding is done by individual tests within each suite.
209+
You can shard by entire test suites (files) using the `--shard-by-suite` flag.
210+
211+
* When sharding by test (default): Distribute individual test cases across
212+
shards. This balances the test load at the test case level. Test cases from
213+
each suite are sliced continuously, which minimizes how often a suite is
214+
split across shards and helps maximize the re-use of suite `setUpAll` and
215+
`tearDownAll` setups.
216+
* When sharding by suite (using `--shard-by-suite`): Distribute entire test suites
217+
across shards. This can be faster for projects with many small test suites as it
218+
avoids loading every suite in every shard. Because test suites are not split,
219+
any `setUpAll` and `tearDownAll` setups within a suite (including shared helper setups imported across suites) run only on the shard executing that suite.
220+
221+
Sharding is particularly useful for distributed testing, where multiple
222+
machines are used to run tests simultaneously. By dividing the test suite into
223+
smaller subsets, you can run tests in parallel across multiple machines, which
224+
can significantly reduce the overall testing time.
225+
226+
### Interaction with Test Filters
227+
228+
The sharding modes interact differently with filters like `--name` or `--tags`:
229+
230+
* When sharding by test case (default), the partition is calculated *after* applying filters. This guarantees matching test cases are distributed as evenly as possible across all shards.
231+
* When sharding by test suite (using `--shard-by-suite`), suite-level annotations (such as `@TestOn` platform selectors and suite-level `@Tags` at the top of a file) are evaluated before sharding to filter out non-matching suites. However, test filters which apply to individual test cases or groups (such as `--name`) will be reflected in the distribution when sharding by test case, but not when sharding by test suite. Test invocations with an uneven number of matching test cases between test suites can cause uneven workloads across the shards.
213232

214233
### Test concurrency
215234

‎pkgs/test/test/runner/runner_test.dart‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ $_runtimeCompilers
8484
(defaults to "$_defaultConcurrency")
8585
--total-shards The total number of invocations of the test runner being run.
8686
--shard-index The index of this test runner invocation (of --total-shards).
87+
--shard-by-suite Distribute entire test suites (_test.dart files) across shards instead of individual tests.
8788
--timeout The default test timeout. For example: 15s, 2x, none
8889
(defaults to "30s")
8990
--suite-load-timeout The timeout for loading a test suite. Loading the test suite includes compiling the test suite. For example: 15s, 2m, none

0 commit comments

Comments
 (0)