Balance tool_integration_tests subshards by expected duration by jmagman · Pull Request #195628 · flutter/flutter

jmagman · 2026-10-29T21:04:54Z

Tool integration.shard is currently split across subshards by file count. Each file test duration ranges from under a second to 10.5 minutes (the longest is swift_package_manager_test). Over the last 3 postsubmit runs Mac subshards took 4.9 to 19.9 minutes and the Windows subshard 2.0 to 24.9. A 2 minute test is a waste of a bot, since the setup and teardown takes much longer than that.

Split by duration instead. With the current shard counts every Mac subshard comes out near 12 minutes, Linux near 13, Windows near 15.

Fixes #185013 (second option).
I'd like to move more of the stand-alone tool tests from the devicelab into this shard (example #195609) to preserve Macs in the CI pool so this balancing will let us add more without thinking too hard about it.

Where do these numbers come from?

The shard already runs dart test with --file-reporter=json (collectMetrics: true) and runDartTest uploads every test result, with its duration, to LUCI ResultDB (this was news to me).

The new update_tool_integration_test_durations reads those back. It asks Buildbucket for the tool_integration_tests builders and their recent successful builds, then sums per-test durations by test file. No auth is needed to run this script. It only writes files present in the tree. Files CI hasn't run yet keep a hand-entered value or get the host median.

A new test in dev/bots/test (runs in framework_tests_misc on all hosts) fails when the json and the files in integration.shard disagree. Adding a test file without a json entry gives:

macos lists exactly the files in integration.shard [E]
  Not in "macos" of dev/bots/tool_integration_test_durations.json:
    test/integration.shard/my_new_feature_test.dart
  Either run:
    dart dev/bots/update_tool_integration_test_durations.dart
  or edit dev/bots/tool_integration_test_durations.json by hand. Each host needs an entry; use the
  test's wall time in seconds, or 0 on hosts where it does not run.

Deleted or renamed files get the matching "in the JSON but not in the tree" message.

A wrong number only affects balance. A missing or unparseable json fails the shard and points at the script.

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

gemini-code-assist

Code Review

This pull request introduces weighted subsharding for integration tests in the Flutter tools, replacing the previous index-based approach. It adds logic to load test durations from a JSON file, implements a weight-based partitioning algorithm, and includes new test coverage for these utilities. The reviewer provided feedback on improving code readability by avoiding variable shadowing and suggested optimizing HTTP performance by using a shared client.

bkonyi

Nice! Thanks for picking this up! This fell off my radar... 😅

Do we expect that we'll need to rerun the duration collection script periodically, outside of adding new tests? If a new test is created and we choose the median as the weight, should we be regenerating the durations file after a few runs to ensure we're using a valid weight?

bkonyi · 2026-10-30T15:33:57Z

+  }
+
+  final result = <String, Map<String, int>>{};
+  for (final MapEntry<String, List<String>> entry in buildersByHost.entries) {


Nit: we should use object destructuring here

for (final MapEntry(:key, :value) in buildersByHost.entries) { // ... }

bkonyi · 2026-10-30T15:35:29Z

+    for (final String builder in entry.value..sort()) {
+      stderr.writeln('Fetching $builds builds of $builder');
+      for (final String invocation in await _recentInvocations(builder, builds)) {
+        for (final MapEntry<String, int> fileDuration in (await _fileDurations(


Ubernit: consider placing the result of await _fileDurations(...) in a local.

bkonyi · 2026-10-30T15:36:38Z

+
+final http.Client _client = http.Client();
+
+Future<void> main(List<String> arguments) async {


Nit: Consider breaking main's body into multiple helper methods for readability.

bkonyi · 2026-10-30T15:37:52Z

+    }
+  }
+
+  final result = <String, Map<String, int>>{};


Consider creating typedefs for nested collection type parameters so it's clear what each key/value represents.

bkonyi · 2026-10-30T15:38:51Z

+        ..sort();
+  final Map<String, Map<String, int>> previous = _readExisting(output);
+  final sampledAnywhere = <String>{for (final Map<String, int> m in result.values) ...m.keys};
+  for (final MapEntry<String, Map<String, int>> host in result.entries) {


Nit: consider using object destructuring here.

jmagman · 2026-10-30T20:30:54Z

Do we expect that we'll need to rerun the duration collection script periodically, outside of adding new tests?

I didn't have a plan for that, but I figured even a very stale number would be closer than the no-weight situation we have now.

If a new test is created and we choose the median as the weight, should we be regenerating the durations file after a few runs to ensure we're using a valid weight?

Probably, but again I figured meh it'll get re-weighted when the next test is added. If we notice the shards getting imbalanced it's easy enough to run the script again.

bkonyi

Fair enough, just figured I'd ask :)

LGTM!

…r#195628) Tool `integration.shard` is currently split across subshards by file count. Each file test duration ranges from under a second to 10.5 minutes (the longest is `swift_package_manager_test`). Over the last 3 postsubmit runs Mac subshards took 4.9 to 19.9 minutes and the Windows subshard 2.0 to 24.9. A 2 minute test is a waste of a bot, since the setup and teardown takes much longer than that. Split by duration instead. With the current shard counts every Mac subshard comes out near 12 minutes, Linux near 13, Windows near 15. Fixes flutter#185013 (second option). I'd like to move more of the stand-alone tool tests from the devicelab into this shard (example flutter#195609) to preserve Macs in the CI pool so this balancing will let us add more without thinking too hard about it. ### Where do these numbers come from? The shard already runs `dart test` with `--file-reporter=json` (`collectMetrics: true`) and `runDartTest` uploads every test result, with its duration, to LUCI ResultDB (this was news to me). The new `update_tool_integration_test_durations` reads those back. It asks Buildbucket for the `tool_integration_tests` builders and their recent successful builds, then sums per-test durations by test file. No auth is needed to run this script. It only writes files present in the tree. Files CI hasn't run yet keep a hand-entered value or get the host median. A new test in dev/bots/test (runs in `framework_tests_misc` on all hosts) fails when the json and the files in `integration.shard` disagree. Adding a test file without a json entry gives: ``` macos lists exactly the files in integration.shard [E] Not in "macos" of dev/bots/tool_integration_test_durations.json: test/integration.shard/my_new_feature_test.dart Either run: dart dev/bots/update_tool_integration_test_durations.dart or edit dev/bots/tool_integration_test_durations.json by hand. Each host needs an entry; use the test's wall time in seconds, or 0 on hosts where it does not run. ``` Deleted or renamed files get the matching "in the JSON but not in the tree" message. A wrong number only affects balance. A missing or unparseable json fails the shard and points at the script. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

Roll Flutter from fcfb4f203eae to 934720261be0 (38 revisions) flutter/flutter@fcfb4f2...9347202 2026-11-03 [email protected] Roll Skia from 5bceaa033999 to c733ad3e37cb (5 revisions) (flutter/flutter#195872) 2026-11-03 [email protected] Roll Skia from 34d2bf479385 to 5bceaa033999 (9 revisions) (flutter/flutter#195864) 2026-11-03 [email protected] fix: prefer pointer events over semantics tap for mouse clicks on web (flutter/flutter#191529) 2026-11-03 [email protected] [VPAT] Expose a semantics text field value while it is not being edited (flutter/flutter#195445) 2026-11-03 [email protected] [Impeller] Make a GL context current before constructing an AiksContext without a surface (flutter/flutter#195629) 2026-11-03 [email protected] [flutter_tools] Eliminate globals.dart from project.dart (flutter/flutter#194066) 2026-11-03 [email protected] [tool] Complete context runner elimination and migrate AppContext usages to ToolContext (flutter/flutter#190797) 2026-11-03 [email protected] [web] Unify font collections between CanvasKit and Skwasm (flutter/flutter#195539) 2026-11-03 [email protected] [Android] Replay predictive back state for retained engines (flutter/flutter#192286) 2026-11-03 [email protected] [vulkan]: Adds depth/stencil synchronization for subpasses (flutter/flutter#195689) 2026-11-03 [email protected] [web] Honor requested upscaling in instantiateImageCodecWithSize and ImageDescriptor.instantiateCodec (flutter/flutter#195836) 2026-11-03 [email protected] Roll Skia from 866b61ca92c9 to 34d2bf479385 (9 revisions) (flutter/flutter#195848) 2026-11-03 [email protected] Roll Fuchsia Test Scripts from xl1LWBuppzu-BnOzV... to B6zarykZp6PKWRueg... (flutter/flutter#195847) 2026-11-03 [email protected] Remove unused ffi (flutter/flutter#195710) 2026-11-02 [email protected] Revert: [Android] Let the raster thread use every core of a chip without little cores (flutter/flutter#195838) 2026-11-02 [email protected] Run 10 impeller iOS benchmarks as one devicelab task (flutter/flutter#195661) 2026-11-02 [email protected] Use named uniforms instead of setFloat in StretchEffect (flutter/flutter#195372) 2026-11-02 [email protected] [flutter_tools] Report when the browser disconnects during a web test run (flutter/flutter#192198) 2026-11-02 [email protected] fix: Resolve memory leak in image_list example (flutter/flutter#189526) 2026-11-02 [email protected] Only run the common local engine configurations in presubmit (flutter/flutter#195688) 2026-11-02 [email protected] Update gcloud_version to version [email protected] (flutter/flutter#195769) 2026-11-02 [email protected] [Impeller] Order the Vulkan copy-texture-to-texture layout transition after the copy (flutter/flutter#195814) 2026-11-02 [email protected] [Impeller] Order the Vulkan ResizeTexture layout transition after its blit (flutter/flutter#195721) 2026-11-02 [email protected] [web] Fix CanvasKit fragment shader uniforms being freed or mutated after recording (flutter/flutter#195718) 2026-11-02 [email protected] Roll Fuchsia Linux SDK from MqypI6RNyVZ_FLUE-... to T_-6-6X1lDMghhAvX... (flutter/flutter#195809) 2026-11-02 [email protected] [Android] Log why Impeller falls back to OpenGLES (flutter/flutter#194764) 2026-11-02 [email protected] Roll Skia from 3f35fb6e591b to 866b61ca92c9 (11 revisions) (flutter/flutter#195802) 2026-11-02 [email protected] Roll Skia from 3f35fb6e591b to 866b61ca92c9 (11 revisions) (flutter/flutter#195802) 2026-11-02 [email protected] Only run the common local engine configurations in presubmit (flutter/flutter#195688) 2026-11-02 [email protected] Balance tool_integration_tests subshards by expected duration (flutter/flutter#195628) 2026-11-02 [email protected] [web_paragraph] Map code units without a cluster to a cluster (flutter/flutter#195384) 2026-11-02 [email protected] [web] Hide synthetic autofill inputs from assistive technology (flutter/flutter#195618) 2026-11-02 [email protected] Roll Packages from 3742a4e to 3f34a6f (8 revisions) (flutter/flutter#195795) 2026-11-02 [email protected] [web] Unskip web tests that pass on CanvasKit and Skwasm: platform_view_test, route_notification_messages_test (flutter/flutter#195605) 2026-11-02 [email protected] Roll Skia from e5ad3868b8db to 3f35fb6e591b (1 revision) (flutter/flutter#195789) 2026-11-02 [email protected] Roll Skia from f5215e74aa16 to e5ad3868b8db (3 revisions) (flutter/flutter#195779) 2026-11-02 [email protected] Roll Skia from e43478a2f9f8 to f5215e74aa16 (1 revision) (flutter/flutter#195776) 2026-11-01 [email protected] Roll Fuchsia Linux SDK from kW0A0xxNkqf38ljm4... to MqypI6RNyVZ_FLUE-... (flutter/flutter#195768) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose ...

…lutter#195863) Extends the duration-weighted subsharding from flutter#195628 to `commands.shard` and `web.shard`. The durations live in `tool_test_durations.json`, regenerated by `update_tool_test_durations.dart`, and `tool_test_durations_test.dart` keeps it in sync with the tree. The fallback for a file with no sample is the median of its own shard. Both shards are currently split in two on Windows and are lopsided by count. `widget_preview_test` and `create_test` are 18 of the 23 minutes of commands.shard on Windows, and `hot_restart_chrome_test` is 8 of the 22 in web.shard. Windows test times: shard | by count (today) | by weight (this PR) -- | -- | -- tool_tests_commands | 0.4 / 22.9 min | 11.7 / 11.7 web_tool_tests | 15.5 / 6.5 min | 11.1 / 10.8 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

…utter#195609) `ios_app_with_extensions_test` runs `flutter build ios` on `dev/integration_tests/ios_app_with_extensions` and checks that the tool tries to build the watchOS companion target. It needs Xcode but nothing else from devicelab and no simulator. The test itself takes ~1 minute, but the builder occupied a Mac host for ~4 minutes because of checkout, CIPD dependencies, result upload, etc. Moving it to `integration.shard` adds that one minute to one of the existing Mac tool_integration_tests shards, which currently spend 6-21 minutes in their test step, and removes the separate bot run. Delete the devicelab target. Same pattern as flutter#142272. This will need to be cherry-picked to the release branches since the builder infra config will be missing. Part of the work to reduce the number of Mac bots in CI. Rebalancing the integration test shard by time is being worked on in flutter#195628 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

…r#196243) The ten `Windows tool_integration_tests shards` are balanced at 14.8 minutes of tests each as of flutter#195628, which with 4-5 minutes of bot setup makes 18-20 minute builders. Fifteen shards bring the test step to about 10 minutes and the builder under 15 min. Another attempt at flutter#189292 but now they are balanced post- flutter#195628. Dropped the denominator from the build name so we don't need to have the `bringup` thrash just to add or remove shards. See flutter#189040. The old ten _N_10 targets will be removed in a follow-up once these are green in post-submit. Fixes flutter#189290 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

The web unit test shards split packages/flutter/test by file count after a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit shards took 12.7 to 24.0 minutes of tests. Split by measured duration instead, the way flutter#195628 and flutter#195863 do for the tool shards. Durations live in dev/bots/web_test_durations.json, keyed by renderer and repository-relative file, regenerated by update_web_test_durations.dart from ResultDB. _runFlutterWebTest now reports its results to ResultDB so that data exists; this seed was measured from the reporter output of one run. web_test_durations_test keeps the file in sync with the tree. The flutter_web_plugins and flutter_driver web tests are part of the partition instead of an extra on the last shard, each kept together as one unit so they compile once. Predicted from the seed: eight canvaskit shards of 16.6 minutes each instead of 12.7 to 24.0, and two skwasm shards of 10.2. The Buildbucket and ResultDB code shared with update_tool_test_durations.dart moves to test_durations.dart.

These overrides date from when the shards ran 30-45 minutes. After flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the 30 minute default is enough and a hang fails sooner.

The web unit test shards split packages/flutter/test by file count after a fixed-seed shuffle, so balance is luck: the last prod run's canvaskit shards took 12.7 to 24.0 minutes of tests. Split by measured duration instead, the way flutter#195628 and flutter#195863 do for the tool shards. Durations live in dev/bots/web_test_durations.json, keyed by renderer and repository-relative file, regenerated by update_web_test_durations.dart from ResultDB. _runFlutterWebTest now reports its results to ResultDB so that data exists; this seed was measured from the reporter output of one run. web_test_durations_test keeps the file in sync with the tree. The flutter_web_plugins and flutter_driver web tests are part of the partition instead of an extra on the last shard, each kept together as one unit so they compile once. Predicted from the seed: eight canvaskit shards of 16.6 minutes each instead of 12.7 to 24.0, and two skwasm shards of 10.2. The Buildbucket and ResultDB code shared with update_tool_test_durations.dart moves to test_durations.dart.

…6277) These overrides date from when the shards ran 30-45 minutes. After flutter#195628, flutter#195863, and flutter#195871 their test steps take 9-20 minutes, so the 30 minute default is enough and a hang fails sooner. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

Balance tool_integration_tests subshards by expected duration

b378d2b

jmagman added the CICD Run CI/CD label Oct 29, 2026

jmagman mentioned this pull request Oct 29, 2026

[chore] Split up long-running tool_integration_tests to have more balanced sharding #185013

Closed

jmagman marked this pull request as ready for review October 30, 2026 00:16

jmagman requested review from bkonyi and jtmcdole October 30, 2026 00:17

gemini-code-assist Bot reviewed Oct 30, 2026

View reviewed changes

Comment thread dev/bots/test/tool_integration_test_durations_test.dart Resolved

Comment thread dev/bots/update_tool_integration_test_durations.dart Resolved

review edits

ca27125

jmagman mentioned this pull request Oct 30, 2026

Move ios_app_with_extensions_test into the tool integration shard #195609

Merged

10 of 11 tasks

bkonyi reviewed Oct 30, 2026

View reviewed changes

nits

67cbc9d

bkonyi approved these changes Nov 2, 2026

View reviewed changes

jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 2, 2026

auto-submit Bot added this pull request to the merge queue Nov 2, 2026

Merged via the queue into flutter:master with commit 3f19e89 Nov 2, 2026
23 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Nov 2, 2026

jmagman deleted the shard-weight branch November 2, 2026 20:08

jmagman mentioned this pull request Nov 3, 2026

Balance tool_tests_commands and web_tool_tests subshards by duration #195863

Merged

11 tasks done

engine-flutter-autoroll mentioned this pull request Nov 3, 2026

Roll Flutter from fcfb4f203eae to 934720261be0 (38 revisions) flutter/packages#13540

Merged

jmagman added the c: contributor-productivity Team-specific productivity, code health, technical debt. label Nov 3, 2026

jmagman mentioned this pull request Nov 9, 2026

Increase number of 'Windows tool_integration_tests' subshards #196243

Merged

11 tasks done

jmagman mentioned this pull request Nov 10, 2026

Balance web_canvaskit_tests and web_skwasm_tests subshards by duration #196260

Open

11 tasks done

jmagman mentioned this pull request Nov 10, 2026

Drop test_timeout_secs from shards that no longer need it #196277

Merged

10 of 11 tasks

Navigation Menu

Repository navigation

Balance tool_integration_tests subshards by expected duration - #195628

Balance tool_integration_tests subshards by expected duration#195628
auto-submit[bot] merged 3 commits into
flutter:masterfrom
jmagman:shard-weight

jmagman commented Oct 29, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

bkonyi left a comment

Uh oh!

bkonyi Oct 30, 2026

Uh oh!

bkonyi Oct 30, 2026

Uh oh!

bkonyi Oct 30, 2026

Uh oh!

bkonyi Oct 30, 2026

Uh oh!

bkonyi Oct 30, 2026

Uh oh!

jmagman commented Oct 30, 2026

Uh oh!

bkonyi left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		final http.Client _client = http.Client();

		Future<void> main(List<String> arguments) async {

Conversation

jmagman commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Where do these numbers come from?

Pre-launch Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

bkonyi left a comment

Choose a reason for hiding this comment

Uh oh!

bkonyi Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

bkonyi Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

bkonyi Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

bkonyi Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

bkonyi Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

jmagman commented Oct 30, 2026

Uh oh!

bkonyi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmagman commented Oct 29, 2026 •

edited

Loading