Repository navigation
[packages] Add repo tooling support for custom sharding #189278
Description
Per discussion in flutter/packages#12150, the material_ui tests are quite slow on (non-WASM) web, such that just material_ui + material_ui/example are too long to be a reasonable shard together (~40 mins).
Currently the way sharding works in flutter/packages is that the tool finds every package that needs to be run in a given PR—the problematic case is anything that runs all packages, such as a flutter/flutter -> flutter/packages roll—then takes the alphabetically ordered list and chops it into N equal count pieces. This has always been a suboptimal way to shard, but we've always gotten away with it because we've never had any single package that was this different in timing from everything else (it's about 10x other packages).
We'll need to decide how we want to implement shard control. Ideally I'd like to keep the property that things can rebalance on the fly (although that has been an issue for flake detection in the past, since tests run in completely different shards for PRs that only touch a small number of packages vs those that run all packages), and ideally it should be low maintenance (i.e., not needing to do a bunch of work when adding one new small package).
Something I'm considering is a file that we could feed in (so we'd have one for each test type that needed shard control) that would just be a yaml map of package name to a "test unit" cost, where the default value is 1, which would correspond to a typical test of that type. For instance, for the web Dart unit tests in question, it looks like the baseline fast unit test time is about 30 seconds, so we'd call that a unit. material_ui takes about 26 minutes, so we'd call that 52 units. material_ui/example/ is about 16 minutes, so we'd call it 32 units. We'd only need to list the slow packages in the file, and everything else would be assumed one (although a bunch are zero because they are skipped after sharding; we would need to see if that ends up being an issue or not).
Then we add up all the units, divide that to get a target number of units per shard, and then implement a simple greed algorithm that would probably be good enough since we don't have to balance perfectly. Probably one of:
- Iteratively take the largest-cost package and put in in the currently-emptiest shard.
- likely gets the best balance, but will make it very hard to guess which shard a test is in compared to the current alphabetical approach.
- Take the next package alphabetically that doesn't go over the target cost (probably with a tolerance of, say 10%).
- likely less smooth, but would keep things mostly alphabetical.
If we go this route, we can try some experiments once we set up cost files, and see how the options handle our current needs in practice.
Activity
stuartmorgan-g commentedon Jul 13, 2026
Hm, there's a deeper problem here which is that we shard at the top-level package level, and then expand to sub-packages later for the tests that need that. But in this case, the two things we most need to separate are the same top-level package. This is going to require more thought.
stuartmorgan-g commentedon Jul 13, 2026
I just remembered the other problem, which is that as our CI is currently set up we need sharding to be consistent across different kinds of tests. That's because for things like https://github.com/flutter/packages/blob/ad2eab1b9fa3901d43b2383272ffd43ddb0b55cb/.ci/targets/ios_platform_tests.yaml we want build examples, native analyze, and native test to all run on the same packages so we can benefit from build caches. If we size packages differently in different tests, each step will run on different packages, and it'll be a mess.
So if we do something with weighting, we would need to guarantee that:
elliette commentedon Sep 4, 2026
Status update here:
Adding a new
shardCostargument topackage_sharding(as attempted in flutter/packages#12715, see specifically here) was not sufficient - the CI targets with the newshardCostargument all failed to run in CI with the following error:{ "error": { "code": 3, "message": "generating build from request: builder \"flutter/try/Linux_web web_dart_unit_test_shard_1 master\": property \"package_sharding\": invalid property override" } }Getting this to work requires:
shardCost, which was done in https://flutter-review.googlesource.com/c/recipes/+/82680package_shardingto the allowed overridable properties in the LUCI service config here: https://flutter.googlesource.com/infra/+/refs/heads/main/config/lib/common.star#13Chatted with @eyebrowsoffire about this: it is much preferred to find a solution that does not require 2 above (adding
package_shardingto the overridable properties). Ideally we need a solution that doesn't involve updating the LUCI config every time we need to change how our CI targets behave.For now, I'm going to revert https://flutter-review.googlesource.com/c/recipes/+/82680. Chatted with @Piinks about this, and we're going to workaround this by reducing our web testing suite for
material_uiandcupertino_ui(issue links to come).However, I would like to keep this open because I do think it would be a good to have a longterm solution here for how we can better shard our tests. One idea from @eyebrowsoffire :