Repository navigation
Move ios_app_with_extensions_test into the tool integration shard - #195609
Merged
auto-submit[bot] merged 2 commits intoNov 5, 2026
Merged
Conversation
jmagman
marked this pull request as ready for review
October 29, 2026 19:36
Contributor
There was a problem hiding this comment.
Code Review
This pull request migrates the ios_app_with_extensions_test from a devicelab task to an integration shard test under packages/flutter_tools, removing the old devicelab task configuration from .ci.yaml and TESTOWNERS. Feedback on the new test suggests making the tempDir variable nullable to prevent a LateInitializationError in tearDown if setUp fails, ensuring that the original setup failure is not masked during debugging.
11 tasks done
dbebawy
pushed a commit
to dbebawy/flutter
that referenced
this pull request
Nov 2, 2026
…r#195628) Tool `integration.shard` is currently split across subshards by file count. Each file test duration ranges from under a second to 10.5 minutes (the longest is `swift_package_manager_test`). Over the last 3 postsubmit runs Mac subshards took 4.9 to 19.9 minutes and the Windows subshard 2.0 to 24.9. A 2 minute test is a waste of a bot, since the setup and teardown takes much longer than that. Split by duration instead. With the current shard counts every Mac subshard comes out near 12 minutes, Linux near 13, Windows near 15. Fixes flutter#185013 (second option). I'd like to move more of the stand-alone tool tests from the devicelab into this shard (example flutter#195609) to preserve Macs in the CI pool so this balancing will let us add more without thinking too hard about it. ### Where do these numbers come from? The shard already runs `dart test` with `--file-reporter=json` (`collectMetrics: true`) and `runDartTest` uploads every test result, with its duration, to LUCI ResultDB (this was news to me). The new `update_tool_integration_test_durations` reads those back. It asks Buildbucket for the `tool_integration_tests` builders and their recent successful builds, then sums per-test durations by test file. No auth is needed to run this script. It only writes files present in the tree. Files CI hasn't run yet keep a hand-entered value or get the host median. A new test in dev/bots/test (runs in `framework_tests_misc` on all hosts) fails when the json and the files in `integration.shard` disagree. Adding a test file without a json entry gives: ``` macos lists exactly the files in integration.shard [E] Not in "macos" of dev/bots/tool_integration_test_durations.json: test/integration.shard/my_new_feature_test.dart Either run: dart dev/bots/update_tool_integration_test_durations.dart or edit dev/bots/tool_integration_test_durations.json by hand. Each host needs an entry; use the test's wall time in seconds, or 0 on hosts where it does not run. ``` Deleted or renamed files get the matching "in the JSON but not in the tree" message. A wrong number only affects balance. A missing or unparseable json fails the shard and points at the script. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
gmackall
pushed a commit
to gmackall/flutter
that referenced
this pull request
Nov 3, 2026
…r#195628) Tool `integration.shard` is currently split across subshards by file count. Each file test duration ranges from under a second to 10.5 minutes (the longest is `swift_package_manager_test`). Over the last 3 postsubmit runs Mac subshards took 4.9 to 19.9 minutes and the Windows subshard 2.0 to 24.9. A 2 minute test is a waste of a bot, since the setup and teardown takes much longer than that. Split by duration instead. With the current shard counts every Mac subshard comes out near 12 minutes, Linux near 13, Windows near 15. Fixes flutter#185013 (second option). I'd like to move more of the stand-alone tool tests from the devicelab into this shard (example flutter#195609) to preserve Macs in the CI pool so this balancing will let us add more without thinking too hard about it. ### Where do these numbers come from? The shard already runs `dart test` with `--file-reporter=json` (`collectMetrics: true`) and `runDartTest` uploads every test result, with its duration, to LUCI ResultDB (this was news to me). The new `update_tool_integration_test_durations` reads those back. It asks Buildbucket for the `tool_integration_tests` builders and their recent successful builds, then sums per-test durations by test file. No auth is needed to run this script. It only writes files present in the tree. Files CI hasn't run yet keep a hand-entered value or get the host median. A new test in dev/bots/test (runs in `framework_tests_misc` on all hosts) fails when the json and the files in `integration.shard` disagree. Adding a test file without a json entry gives: ``` macos lists exactly the files in integration.shard [E] Not in "macos" of dev/bots/tool_integration_test_durations.json: test/integration.shard/my_new_feature_test.dart Either run: dart dev/bots/update_tool_integration_test_durations.dart or edit dev/bots/tool_integration_test_durations.json by hand. Each host needs an entry; use the test's wall time in seconds, or 0 on hosts where it does not run. ``` Deleted or renamed files get the matching "in the JSON but not in the tree" message. A wrong number only affects balance. A missing or unparseable json fails the shard and points at the script. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
jmagman
force-pushed
the
ios_app_with_extensions_test
branch
from
November 5, 2026 07:50
03db75c to
c1d9453
Compare
vashworth
approved these changes
Nov 5, 2026
chingjun
added a commit
to chingjun/flutter
that referenced
this pull request
Nov 5, 2026
…ealm Mac tool_integration_tests_10 at 523: ios_app_with_extensions_test (in the integration shard since master's flutter#195609) runs `flutter build ios`, which downloads the iOS engine artifacts at run time, and the tool under test found an empty bin/cache/engine.realm in its SDK facsimile (bin/internal:engine_realm wrote one), so it asked the release bucket for a presubmit's engine -- a 404; the checkout's own tool reads the realm Cocoon's FLUTTER_REALM put in bin/cache/engine.realm (flutter_archives_v2) through bin/internal/update_engine_version.sh and looks under it. @engine_version (repositories.bzl) now also records ENGINE_REALM -- FLUTTER_REALM when set, else the checkout's bin/cache/engine.realm (watched), else empty -- and the facsimile's engine.realm is that. Checked from the workstation: empty without, `flutter_archives_v2` with FLUTTER_REALM in the environment, the repository re-evaluating on the change.
chingjun
added a commit
to chingjun/flutter
that referenced
this pull request
Nov 6, 2026
The integration shard's facsimile ran master's new ios_app_with_extensions_test (flutter#195609) on a Mac: `flutter build ios --no-codesign --release` of dev/integration_tests/ios_app_with_extensions, whose project the facsimile did not hold ("Target file lib/main.dart not found"; Mac tool_integration_tests_10 at 94e964f, after 525–527 had taken the same test through the engine download). The file builds for iOS through Xcode, which is what _INTEGRATION_APPLE_IOS gathers: it moves there. The Mac tool_integration_tests shard that holds it runs it as before through dev/bots' own `dart test` in the checkout (a file no Linux-capable suite covers falls back to the legacy path, as the other apple files do), and the apple shard -- a Mac's, by name -- runs it from its facsimile: the project is in _APPLE_DATA (//dev/integration_tests/ios_app_with_extensions:project, new BUILD.bazel). On Linux it would only report itself skipped, as the others.
auto-submit Bot
pushed a commit
to flutter/packages
that referenced
this pull request
Nov 6, 2026
Roll Flutter from 3623d2d34543 to 09175cdeac89 (61 revisions) flutter/flutter@3623d2d...09175cd 2026-11-06 [email protected] Roll Skia from ae35b7b2221f to 793b8ac73d4b (3 revisions) (flutter/flutter#196091) 2026-11-06 [email protected] [web] Rasterize scenes at the size passed to `FlutterView.render` (flutter/flutter#196052) 2026-11-06 [email protected] Roll Skia from 4aa043a81e0c to ae35b7b2221f (1 revision) (flutter/flutter#196086) 2026-11-06 [email protected] Roll Fuchsia Linux SDK from Fqe3UlLwAohsPbsCS... to KIc0XZbhLmvDpTluZ... (flutter/flutter#196085) 2026-11-06 [email protected] Allow a zero scale change in InteractiveViewer (flutter/flutter#194949) 2026-11-06 [email protected] Roll Skia from dd5c2553a856 to 4aa043a81e0c (2 revisions) (flutter/flutter#196079) 2026-11-06 [email protected] [web] Keep ImageFilter.blur bounds in equality and description (flutter/flutter#196013) 2026-11-06 [email protected] iOS: Remove unused PlatformViewIOS setup from FlutterPlatformViewsTest (flutter/flutter#196033) 2026-11-06 [email protected] Serialize LoggerTests that mutate the shared Logger (flutter/flutter#196028) 2026-11-06 [email protected] Use reverseDuration for the AnimatedCrossFade size animation (flutter/flutter#194860) 2026-11-05 [email protected] Roll Skia from a5aacd103e68 to dd5c2553a856 (8 revisions) (flutter/flutter#196073) 2026-11-05 [email protected] Fix ListWheelScrollView null check error when it reads itemIndex before content dimensions are set (flutter/flutter#194859) 2026-11-05 [email protected] Allow zero cross axis extent in SliverGridDelegateWithMaxCrossAxisExtent (flutter/flutter#194858) 2026-11-05 [email protected] Revert: Allow color delta tolerance in in-memory fuzzy image comparator (flutter/flutter#196067) 2026-11-05 [email protected] Revert: Mitigate snapshot race in 3D transform test by reducing loop density (flutter/flutter#196066) 2026-11-05 [email protected] Delegate createStream in ResizeImage and ScrollAwareImageProvider (flutter/flutter#194756) 2026-11-05 [email protected] [iOS] Introduce FlutterShell and move thread host and profiler (flutter/flutter#195388) 2026-11-05 [email protected] [web] Support FragmentShader.getImageSampler on CanvasKit and Skwasm (flutter/flutter#195854) 2026-11-05 [email protected] Deprecates setFloat API (flutter/flutter#195505) 2026-11-05 [email protected] [flutter_tools] Remove public GlobalToolContext and globals from runner (flutter/flutter#195561) 2026-11-05 [email protected] Defer TLHC PlatformViewWrapper frame post and avoid initial 1x1 buffer resize (flutter/flutter#195861) 2026-11-05 [email protected] Roll Skia from b5abed66da86 to a5aacd103e68 (17 revisions) (flutter/flutter#196054) 2026-11-05 [email protected] Made `FlutterWindowControllerSizeTest.SizedToContentNotResizable` more robust (flutter/flutter#195845) 2026-11-05 [email protected] Update docs for contributor ladder (flutter/flutter#196011) 2026-11-05 [email protected] [Impeller] Use the transfer stage for the Vulkan readback barrier (flutter/flutter#194991) 2026-11-05 [email protected] Remove Mac mac_unopt builder (flutter/flutter#196027) 2026-11-05 [email protected] Updated Instructions for Determining Gradle Version for `gradle_dists` in CIPD (flutter/flutter#193310) 2026-11-05 [email protected] Move ios_app_with_extensions_test into the tool integration shard (flutter/flutter#195609) 2026-11-05 [email protected] [web] Respect Uint8List view bounds when detecting animated WebP/GIF (flutter/flutter#196010) 2026-11-05 [email protected] Add platform_views_layout to build_tests so iOS/Android build breaks are caught in presubmit (flutter/flutter#192985) 2026-11-05 [email protected] [flutter_tools] Detect macOS TCC permission crashes and provide diagnostic guidance (flutter/flutter#192432) 2026-11-05 [email protected] Roll Packages from 670b30e to 0478162 (15 revisions) (flutter/flutter#196047) 2026-11-05 [email protected] [web] Support Skwasm rendering without bitmap capture (flutter/flutter#195103) 2026-11-05 [email protected] [flutter_tools] Reland fix for local CanvasKit asset paths on Windows (flutter/flutter#195770) 2026-11-05 [email protected] Roll Skia from f96be51af95d to b5abed66da86 (14 revisions) (flutter/flutter#196017) 2026-11-05 [email protected] Split chrome web engine test drone (flutter/flutter#195873) 2026-11-05 [email protected] Roll Dart SDK from c9153304136a to 6623b84628f3 (1 revision) (flutter/flutter#196026) 2026-11-05 [email protected] Roll ICU from 4fe960221710 to a0ea93436c52 (6 revisions) (flutter/flutter#196025) 2026-11-05 [email protected] Add more guidance in README and AGENTS for contributing to flutter_tools (flutter/flutter#195998) 2026-11-05 [email protected] Roll Fuchsia Linux SDK from p0HDXLISPIElJfSz7... to Fqe3UlLwAohsPbsCS... (flutter/flutter#196019) 2026-11-05 [email protected] Roll Dart SDK from e362a23e9738 to c9153304136a (18 revisions) (flutter/flutter#196014) 2026-11-05 [email protected] [Impeller] Fix dilation input bounds when clipping saveLayers (flutter/flutter#195652) 2026-11-05 [email protected] Add `variantOf` to the M3E migration checklist (flutter/flutter#195099) 2026-11-05 [email protected] [flutter_tools] Eliminate globals from remaining subsystem files (flutter/flutter#195555) 2026-11-05 [email protected] Fix gap flipping when dragging variable height reorderable items (flutter/flutter#194478) 2026-11-05 [email protected] [A11y] Send un-merged SemanticsNodes in the same update that first references them (flutter/flutter#195829) ...
chingjun
added a commit
to chingjun/flutter
that referenced
this pull request
Nov 12, 2026
…ealm Mac tool_integration_tests_10 at 523: ios_app_with_extensions_test (in the integration shard since master's flutter#195609) runs `flutter build ios`, which downloads the iOS engine artifacts at run time, and the tool under test found an empty bin/cache/engine.realm in its SDK facsimile (bin/internal:engine_realm wrote one), so it asked the release bucket for a presubmit's engine -- a 404; the checkout's own tool reads the realm Cocoon's FLUTTER_REALM put in bin/cache/engine.realm (flutter_archives_v2) through bin/internal/update_engine_version.sh and looks under it. @engine_version (repositories.bzl) now also records ENGINE_REALM -- FLUTTER_REALM when set, else the checkout's bin/cache/engine.realm (watched), else empty -- and the facsimile's engine.realm is that. Checked from the workstation: empty without, `flutter_archives_v2` with FLUTTER_REALM in the environment, the repository re-evaluating on the change.
chingjun
added a commit
to chingjun/flutter
that referenced
this pull request
Nov 12, 2026
The integration shard's facsimile ran master's new ios_app_with_extensions_test (flutter#195609) on a Mac: `flutter build ios --no-codesign --release` of dev/integration_tests/ios_app_with_extensions, whose project the facsimile did not hold ("Target file lib/main.dart not found"; Mac tool_integration_tests_10 at 94e964f, after 525–527 had taken the same test through the engine download). The file builds for iOS through Xcode, which is what _INTEGRATION_APPLE_IOS gathers: it moves there. The Mac tool_integration_tests shard that holds it runs it as before through dev/bots' own `dart test` in the checkout (a file no Linux-capable suite covers falls back to the legacy path, as the other apple files do), and the apple shard -- a Mac's, by name -- runs it from its facsimile: the project is in _APPLE_DATA (//dev/integration_tests/ios_app_with_extensions:project, new BUILD.bazel). On Linux it would only report itself skipped, as the others.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ios_app_with_extensions_testrunsflutter build iosondev/integration_tests/ios_app_with_extensionsand checks that the tool tries to build the watchOS companion target. It needs Xcode but nothing else from devicelab and no simulator.The test itself takes ~1 minute, but the builder occupied a Mac host for ~4 minutes because of checkout, CIPD dependencies, result upload, etc. Moving it to
integration.shardadds that one minute to one of the existing Mac tool_integration_tests shards, which currently spend 6-21 minutes in their test step, and removes the separate bot run.Delete the devicelab target. Same pattern as #142272. This will need to be cherry-picked to the release branches since the builder infra config will be missing.
Part of the work to reduce the number of Mac bots in CI. Rebalancing the integration test shard by time is being worked on in #195628
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.