Skip to content

Balance tool_tests_commands and web_tool_tests subshards by duration - #195863

Merged
auto-submit[bot] merged 2 commits into
flutter:masterfrom
jmagman:tool_commands_weights
Nov 4, 2026
Merged

auto-submit[bot] merged 2 commits into
flutter:masterfrom
jmagman:tool_commands_weights

Conversation

@jmagman

@jmagman jmagman commented Nov 3, 2026

Copy link
Copy Markdown
Member

Extends the duration-weighted subsharding from #195628 to commands.shard and web.shard.

The durations live in tool_test_durations.json, regenerated by update_tool_test_durations.dart, and tool_test_durations_test.dart keeps it in sync with the tree. The fallback for a file with no sample is the median of its own shard.

Both shards are currently split in two on Windows and are lopsided by count.
widget_preview_test and create_test are 18 of the 23 minutes of commands.shard on Windows, and hot_restart_chrome_test is 8 of the 22 in web.shard.

Windows test times:

shard by count (today) by weight (this PR)
tool_tests_commands 0.4 / 22.9 min 11.7 / 11.7
web_tool_tests 15.5 / 6.5 min 11.1 / 10.8

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

@jmagman jmagman added the CICD Run CI/CD label Nov 3, 2026
@jmagman

jmagman commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the tool test execution in dev/bots/test.dart to use a unified _selectWeightedToolTests helper for selecting weighted tests across the 'commands', 'web', and 'integration' shards. Consequently, tool_integration_test_durations.json is renamed to tool_test_durations.json and expanded to include durations for all three shards, and the script to update these durations is updated and renamed to update_tool_test_durations.dart. Feedback suggests making the regular expressions for matching builder names in update_tool_test_durations.dart more robust and consistent across shards by matching both Mac and Mac_arm64 using Mac(?:_arm64)?.

@jmagman
jmagman force-pushed the tool_commands_weights branch from f376a0a to 7af38b3 Compare November 3, 2026 04:30
@jmagman
jmagman marked this pull request as ready for review November 3, 2026 05:10

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request extends the weighted test subsharding mechanism to cover commands and web tool tests in addition to integration tests. It refactors the test runner to use a shared helper for selecting weighted tests, updates the duration tracking files and synchronization tests, and adapts the duration regeneration script to handle multiple shards. Feedback suggests reordering the keys in _hostByPrefix within the update script to place 'Mac_arm64' before 'Mac', which prevents potential regex backtracking issues during matching.

@jmagman
jmagman requested a review from bkonyi November 3, 2026 19:27
@jmagman jmagman added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Nov 3, 2026

@bkonyi bkonyi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@jmagman jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 4, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Nov 4, 2026
Merged via the queue into flutter:master with commit 1747c0a Nov 4, 2026
29 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Nov 4, 2026
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Nov 4, 2026
flutter/flutter@9347202...3623d2d

2026-11-04 [email protected] Roll Skia from 49ec2655bf51 to aaa39bfcd393 (5 revisions) (flutter/flutter#195956)
2026-11-04 [email protected] Roll pub packages (flutter/flutter#195950)
2026-11-04 [email protected] Roll pub packages (flutter/flutter#195948)
2026-11-04 [email protected] local_engine_builds_common out of bringup (flutter/flutter#195862)
2026-11-04 [email protected] [web][a11y] Initialize slider semantics values when enabling (flutter/flutter#195564)
2026-11-04 [email protected] Drop Android SDK, JDK, and Gradle dependencies from framework_tests_misc (flutter/flutter#195869)
2026-11-04 [email protected] Roll Fuchsia Test Scripts from B6zarykZp6PKWRueg... to Jo-ZgQSNDuPsjlZ4s... (flutter/flutter#195937)
2026-11-04 [email protected] Balance tool_tests_commands and web_tool_tests subshards by duration (flutter/flutter#195863)
2026-11-04 [email protected] Roll Fuchsia Linux SDK from T_-6-6X1lDMghhAvX... to p0HDXLISPIElJfSz7... (flutter/flutter#195939)
2026-11-03 [email protected] Roll Skia from 3eeafac94f07 to 49ec2655bf51 (8 revisions) (flutter/flutter#195933)
2026-11-03 [email protected] Roll pub packages (flutter/flutter#195932)
2026-11-03 [email protected] [flutter_tools] Lazily initialize AppleContext dependencies via builder closures (flutter/flutter#191992)
2026-11-03 [email protected] [Android] Upgrade linux_android_emu_vulkan_stable avd_cipd_version to 36.1.9 (build_id:8670032186093418337) (flutter/flutter#195856)
2026-11-03 [email protected] [vulkan]: adds "color attachment read" dst dependency to subpass (flutter/flutter#195714)
2026-11-03 [email protected] [flutter_tools] Stream web dev server proxy responses instead of buffering them (flutter/flutter#194934)
2026-11-03 [email protected] Roll Skia from d80db8f8f856 to 3eeafac94f07 (6 revisions) (flutter/flutter#195926)
2026-11-03 [email protected] [flutter_tools] Add flutter assemble support for tool extension assemble targets (flutter/flutter#194563)
2026-11-03 [email protected] Manual roll Dart SDK from fe609902984b to e362a23e9738 (2 revisions) (flutter/flutter#195914)
2026-11-03 [email protected] feat(flutter_test): add startHover and startHoverAt helper to WidgetTester (flutter/flutter#190630)
2026-11-03 [email protected] [flutter_tools] Remove trailing dots from bug report URLs (flutter/flutter#194132)
2026-11-03 [email protected] Update skills_lint to 0.5.3 (flutter/flutter#195796)
2026-11-03 [email protected] Roll Skia from 58295d932350 to d80db8f8f856 (5 revisions) (flutter/flutter#195900)
2026-11-03 [email protected] [flutter_tools] Make DeviceLogReader.dispose asynchronous (flutter/flutter#195389)
2026-11-03 [email protected] Roll Packages from 3f34a6f to 1df8447 (10 revisions) (flutter/flutter#195895)
2026-11-03 [email protected] Roll Skia from c733ad3e37cb to 58295d932350 (1 revision) (flutter/flutter#195894)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
chingjun added a commit to chingjun/flutter that referenced this pull request Nov 5, 2026
…migration/phase0

The branch was 61 commits behind master. Merged rather than rebased: the
branch's commits are pushed and under review (PR flutter#196005).

Conflicts, resolved by hand:
- dev/bots/test.dart: master's flutter#195863 balances tool_tests_commands and
  web_tool_tests by duration through _selectWeightedToolTests (one
  durations file, dev/bots/tool_test_durations.json, for the three tool
  shards); the Bazel branches of the three shard runners now cover the
  paths it selects, as before over selectIndexOfTotalSubshard's.

Regenerated for master's rolls and renames:
- build_defs/repositories/deps.lock.bzl (gen_deps.dart): the Dart SDK roll
  to c9153304136a and its package revisions, the Fuchsia SDK and test
  scripts.
- build_defs/dart/pub.lock.bzl (gen_pub_lock.dart after `dart pub get` at
  the root and in lib/web_ui): the pubspec changes of master.
- engine/src/flutter/lib/ui/BUILD.bazel (gn_to_bazel): the GLES
  compositing unit test renamed to impeller_gles_without_surface_unittests.cc.
- engine/src/flutter/impeller/BUILD.bazel (gn_to_bazel): run_tests.py's
  Vulkan synchronization-validation list grew by the tests flutter#194989 fixed.
  `gn_to_bazel --check` is current.

Adjusted by hand for master's renames:
- dev/bots/BUILD.bazel and packages/flutter_tools/BUILD.bazel: the durations
  test is tool_test_durations_test over tool_test_durations.json and the
  test files of the commands, integration and web shards
  (`durations_shard_test_files`, was `integration_shard_test_files`).
- dev/bots/test/test_test.dart: the `_bazel` subshard test expects master's
  "Selecting subshard 1 of 3 (N of M tests, estimated weight W)".

Checked: gclient sync clean (Skia f96be51af95d, Dart c9153304136a);
gn_to_bazel 282/282 and --check current; bazel_tools 46/46; bazel_ci_test.py
23; pylint clean; Bazel analysis of //:framework_tests //:tool_tests
//:dart_tests //dev/bots:tests and of ci_host_debug_unopt (+tests),
android_debug, wasm_release, ci_android_release_arm64; `bazel test
//dev/bots:tests` green; dev/bots test_test.dart's Bazel tests green (its
`--dry-run prints every test` test depends on SHARD being set in the
environment, on master as here).

Note: master's flutter#195868 adds the mac_unopt builds as standalone bringup
builders under ci/builders/standalone/ (GN; the converter does not read
that directory). mac_unopt.json itself still carries --bazel.
jmagman added a commit to jmagman/flutter that referenced this pull request Nov 9, 2026
Mac web_tool_tests runs the 18 web.shard files serially in one shard and
takes 23-26 minutes. The shard already picks files by the macos weights
in tool_test_durations.json (flutter#195863), so add two duration-balanced
subshards, 10.5 and 10.2 minutes of tests each, the same shape as
Windows. The names drop the shard count, as flutter#189040 did for Mac
tool_integration_tests, so future resharding does not need a bringup
round. A follow-up removes Mac web_tool_tests once they are green.
ChandraChaan pushed a commit to ChandraChaan/flutter that referenced this pull request Nov 10, 2026
`Mac web_tool_tests` runs the 18 `web.shard` files serially in one shard
and takes 23-26 minutes, which is currently the second-longest Mac
framework shard. The shard already picks files by the macos weights in
`tool_test_durations` as of flutter#195863, so add two duration-balanced
subshards, 10.5 and 10.2 minutes of tests each, same as Windows:

https://github.com/flutter/flutter/blob/236153d39ea761068f6dc0958551eac765866fb7/.ci.yaml#L7149

This will make presubmit faster by reducing the last "long pole"
builders, but will incur the cost of another Mac builder with
setup/teardown times.

Will follow-up to remove `Mac web_tool_tests` once these two are out of
bring-up.


## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
jmagman added a commit to jmagman/flutter that referenced this pull request Nov 10, 2026
The web unit test shards split packages/flutter/test by file count after
a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit
shards took 12.7 to 24.0 minutes of tests. Split by measured duration
instead, the way flutter#195628 and flutter#195863 do for the tool shards.

Durations live in dev/bots/web_test_durations.json, keyed by renderer
and repository-relative file, regenerated by
update_web_test_durations.dart from ResultDB. _runFlutterWebTest now
reports its results to ResultDB so that data exists; this seed was
measured from the reporter output of one run. web_test_durations_test
keeps the file in sync with the tree. The flutter_web_plugins and
flutter_driver web tests are part of the partition instead of an extra
on the last shard, each kept together as one unit so they compile once.

Predicted from the seed: eight canvaskit shards of 16.6 minutes each
instead of 12.7 to 24.0, and two skwasm shards of 10.2.

The Buildbucket and ResultDB code shared with
update_tool_test_durations.dart moves to test_durations.dart.
jmagman added a commit to jmagman/flutter that referenced this pull request Nov 10, 2026
These overrides date from when the shards ran 30-45 minutes. After
flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so
the 30 minute default is enough and a hang fails sooner.
jmagman added a commit to jmagman/flutter that referenced this pull request Nov 10, 2026
The web unit test shards split packages/flutter/test by file count after
a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit
shards took 12.7 to 24.0 minutes of tests. Split by measured duration
instead, the way flutter#195628 and flutter#195863 do for the tool shards.

Durations live in dev/bots/web_test_durations.json, keyed by renderer
and repository-relative file, regenerated by
update_web_test_durations.dart from ResultDB. _runFlutterWebTest now
reports its results to ResultDB so that data exists; this seed was
measured from the reporter output of one run. web_test_durations_test
keeps the file in sync with the tree. The flutter_web_plugins and
flutter_driver web tests are part of the partition instead of an extra
on the last shard, each kept together as one unit so they compile once.

Predicted from the seed: eight canvaskit shards of 16.6 minutes each
instead of 12.7 to 24.0, and two skwasm shards of 10.2.

The Buildbucket and ResultDB code shared with
update_tool_test_durations.dart moves to test_durations.dart.
kthguru pushed a commit to infradise/flutter that referenced this pull request Nov 11, 2026
…6277)

These overrides date from when the shards ran 30-45 minutes. After
flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the
30 minute default is enough and a hang fails sooner.

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [ ] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

c: contributor-productivity Team-specific productivity, code health, technical debt. CICD Run CI/CD

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants