Repository navigation
Balance tool_tests_commands and web_tool_tests subshards by duration - #195863
Conversation
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request refactors the tool test execution in dev/bots/test.dart to use a unified _selectWeightedToolTests helper for selecting weighted tests across the 'commands', 'web', and 'integration' shards. Consequently, tool_integration_test_durations.json is renamed to tool_test_durations.json and expanded to include durations for all three shards, and the script to update these durations is updated and renamed to update_tool_test_durations.dart. Feedback suggests making the regular expressions for matching builder names in update_tool_test_durations.dart more robust and consistent across shards by matching both Mac and Mac_arm64 using Mac(?:_arm64)?.
f376a0a to
7af38b3
Compare
There was a problem hiding this comment.
Code Review
This pull request extends the weighted test subsharding mechanism to cover commands and web tool tests in addition to integration tests. It refactors the test runner to use a shared helper for selecting weighted tests, updates the duration tracking files and synchronization tests, and adapts the duration regeneration script to handle multiple shards. Feedback suggests reordering the keys in _hostByPrefix within the update script to place 'Mac_arm64' before 'Mac', which prevents potential regex backtracking issues during matching.
flutter/flutter@9347202...3623d2d 2026-11-04 [email protected] Roll Skia from 49ec2655bf51 to aaa39bfcd393 (5 revisions) (flutter/flutter#195956) 2026-11-04 [email protected] Roll pub packages (flutter/flutter#195950) 2026-11-04 [email protected] Roll pub packages (flutter/flutter#195948) 2026-11-04 [email protected] local_engine_builds_common out of bringup (flutter/flutter#195862) 2026-11-04 [email protected] [web][a11y] Initialize slider semantics values when enabling (flutter/flutter#195564) 2026-11-04 [email protected] Drop Android SDK, JDK, and Gradle dependencies from framework_tests_misc (flutter/flutter#195869) 2026-11-04 [email protected] Roll Fuchsia Test Scripts from B6zarykZp6PKWRueg... to Jo-ZgQSNDuPsjlZ4s... (flutter/flutter#195937) 2026-11-04 [email protected] Balance tool_tests_commands and web_tool_tests subshards by duration (flutter/flutter#195863) 2026-11-04 [email protected] Roll Fuchsia Linux SDK from T_-6-6X1lDMghhAvX... to p0HDXLISPIElJfSz7... (flutter/flutter#195939) 2026-11-03 [email protected] Roll Skia from 3eeafac94f07 to 49ec2655bf51 (8 revisions) (flutter/flutter#195933) 2026-11-03 [email protected] Roll pub packages (flutter/flutter#195932) 2026-11-03 [email protected] [flutter_tools] Lazily initialize AppleContext dependencies via builder closures (flutter/flutter#191992) 2026-11-03 [email protected] [Android] Upgrade linux_android_emu_vulkan_stable avd_cipd_version to 36.1.9 (build_id:8670032186093418337) (flutter/flutter#195856) 2026-11-03 [email protected] [vulkan]: adds "color attachment read" dst dependency to subpass (flutter/flutter#195714) 2026-11-03 [email protected] [flutter_tools] Stream web dev server proxy responses instead of buffering them (flutter/flutter#194934) 2026-11-03 [email protected] Roll Skia from d80db8f8f856 to 3eeafac94f07 (6 revisions) (flutter/flutter#195926) 2026-11-03 [email protected] [flutter_tools] Add flutter assemble support for tool extension assemble targets (flutter/flutter#194563) 2026-11-03 [email protected] Manual roll Dart SDK from fe609902984b to e362a23e9738 (2 revisions) (flutter/flutter#195914) 2026-11-03 [email protected] feat(flutter_test): add startHover and startHoverAt helper to WidgetTester (flutter/flutter#190630) 2026-11-03 [email protected] [flutter_tools] Remove trailing dots from bug report URLs (flutter/flutter#194132) 2026-11-03 [email protected] Update skills_lint to 0.5.3 (flutter/flutter#195796) 2026-11-03 [email protected] Roll Skia from 58295d932350 to d80db8f8f856 (5 revisions) (flutter/flutter#195900) 2026-11-03 [email protected] [flutter_tools] Make DeviceLogReader.dispose asynchronous (flutter/flutter#195389) 2026-11-03 [email protected] Roll Packages from 3f34a6f to 1df8447 (10 revisions) (flutter/flutter#195895) 2026-11-03 [email protected] Roll Skia from c733ad3e37cb to 58295d932350 (1 revision) (flutter/flutter#195894) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
…migration/phase0 The branch was 61 commits behind master. Merged rather than rebased: the branch's commits are pushed and under review (PR flutter#196005). Conflicts, resolved by hand: - dev/bots/test.dart: master's flutter#195863 balances tool_tests_commands and web_tool_tests by duration through _selectWeightedToolTests (one durations file, dev/bots/tool_test_durations.json, for the three tool shards); the Bazel branches of the three shard runners now cover the paths it selects, as before over selectIndexOfTotalSubshard's. Regenerated for master's rolls and renames: - build_defs/repositories/deps.lock.bzl (gen_deps.dart): the Dart SDK roll to c9153304136a and its package revisions, the Fuchsia SDK and test scripts. - build_defs/dart/pub.lock.bzl (gen_pub_lock.dart after `dart pub get` at the root and in lib/web_ui): the pubspec changes of master. - engine/src/flutter/lib/ui/BUILD.bazel (gn_to_bazel): the GLES compositing unit test renamed to impeller_gles_without_surface_unittests.cc. - engine/src/flutter/impeller/BUILD.bazel (gn_to_bazel): run_tests.py's Vulkan synchronization-validation list grew by the tests flutter#194989 fixed. `gn_to_bazel --check` is current. Adjusted by hand for master's renames: - dev/bots/BUILD.bazel and packages/flutter_tools/BUILD.bazel: the durations test is tool_test_durations_test over tool_test_durations.json and the test files of the commands, integration and web shards (`durations_shard_test_files`, was `integration_shard_test_files`). - dev/bots/test/test_test.dart: the `_bazel` subshard test expects master's "Selecting subshard 1 of 3 (N of M tests, estimated weight W)". Checked: gclient sync clean (Skia f96be51af95d, Dart c9153304136a); gn_to_bazel 282/282 and --check current; bazel_tools 46/46; bazel_ci_test.py 23; pylint clean; Bazel analysis of //:framework_tests //:tool_tests //:dart_tests //dev/bots:tests and of ci_host_debug_unopt (+tests), android_debug, wasm_release, ci_android_release_arm64; `bazel test //dev/bots:tests` green; dev/bots test_test.dart's Bazel tests green (its `--dry-run prints every test` test depends on SHARD being set in the environment, on master as here). Note: master's flutter#195868 adds the mac_unopt builds as standalone bringup builders under ci/builders/standalone/ (GN; the converter does not read that directory). mac_unopt.json itself still carries --bazel.
Mac web_tool_tests runs the 18 web.shard files serially in one shard and takes 23-26 minutes. The shard already picks files by the macos weights in tool_test_durations.json (flutter#195863), so add two duration-balanced subshards, 10.5 and 10.2 minutes of tests each, the same shape as Windows. The names drop the shard count, as flutter#189040 did for Mac tool_integration_tests, so future resharding does not need a bringup round. A follow-up removes Mac web_tool_tests once they are green.
`Mac web_tool_tests` runs the 18 `web.shard` files serially in one shard and takes 23-26 minutes, which is currently the second-longest Mac framework shard. The shard already picks files by the macos weights in `tool_test_durations` as of flutter#195863, so add two duration-balanced subshards, 10.5 and 10.2 minutes of tests each, same as Windows: https://github.com/flutter/flutter/blob/236153d39ea761068f6dc0958551eac765866fb7/.ci.yaml#L7149 This will make presubmit faster by reducing the last "long pole" builders, but will incur the cost of another Mac builder with setup/teardown times. Will follow-up to remove `Mac web_tool_tests` once these two are out of bring-up. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
The web unit test shards split packages/flutter/test by file count after a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit shards took 12.7 to 24.0 minutes of tests. Split by measured duration instead, the way flutter#195628 and flutter#195863 do for the tool shards. Durations live in dev/bots/web_test_durations.json, keyed by renderer and repository-relative file, regenerated by update_web_test_durations.dart from ResultDB. _runFlutterWebTest now reports its results to ResultDB so that data exists; this seed was measured from the reporter output of one run. web_test_durations_test keeps the file in sync with the tree. The flutter_web_plugins and flutter_driver web tests are part of the partition instead of an extra on the last shard, each kept together as one unit so they compile once. Predicted from the seed: eight canvaskit shards of 16.6 minutes each instead of 12.7 to 24.0, and two skwasm shards of 10.2. The Buildbucket and ResultDB code shared with update_tool_test_durations.dart moves to test_durations.dart.
These overrides date from when the shards ran 30-45 minutes. After flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the 30 minute default is enough and a hang fails sooner.
The web unit test shards split packages/flutter/test by file count after a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit shards took 12.7 to 24.0 minutes of tests. Split by measured duration instead, the way flutter#195628 and flutter#195863 do for the tool shards. Durations live in dev/bots/web_test_durations.json, keyed by renderer and repository-relative file, regenerated by update_web_test_durations.dart from ResultDB. _runFlutterWebTest now reports its results to ResultDB so that data exists; this seed was measured from the reporter output of one run. web_test_durations_test keeps the file in sync with the tree. The flutter_web_plugins and flutter_driver web tests are part of the partition instead of an extra on the last shard, each kept together as one unit so they compile once. Predicted from the seed: eight canvaskit shards of 16.6 minutes each instead of 12.7 to 24.0, and two skwasm shards of 10.2. The Buildbucket and ResultDB code shared with update_tool_test_durations.dart moves to test_durations.dart.
…6277) These overrides date from when the shards ran 30-45 minutes. After flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the 30 minute default is enough and a hang fails sooner. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Extends the duration-weighted subsharding from #195628 to
commands.shardandweb.shard.The durations live in
tool_test_durations.json, regenerated byupdate_tool_test_durations.dart, andtool_test_durations_test.dartkeeps it in sync with the tree. The fallback for a file with no sample is the median of its own shard.Both shards are currently split in two on Windows and are lopsided by count.
widget_preview_testandcreate_testare 18 of the 23 minutes of commands.shard on Windows, andhot_restart_chrome_testis 8 of the 22 in web.shard.Windows test times:
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.