Repository navigation
Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 - #192472
Merged
auto-submit[bot] merged 1 commit intoSep 23, 2026
Merged
auto-submit[bot] merged 1 commit into
auto-submit[bot] merged 1 commit into
Conversation
okorohelijah
marked this pull request as ready for review
September 9, 2026 10:11
Contributor
There was a problem hiding this comment.
Code Review
This pull request modifies .ci.yaml by renaming several macOS ARM64 targets to remove the _ios suffix, updating their tags to include hostonly and arm64 instead of ios, and adding a Ruby dependency to the flutter_gallery_ios__compile target. There are no review comments, and I have no feedback to provide.
Part of flutter#186414 The iOS compile benchmarks (basic_material_app_ios__compile, flutter_gallery_ios__compile, hello_world_ios__compile and imitation_game_flutter__compile) only run `flutter build ios`; they never launch an app on the attached iPhone. Running them on `Mac_arm64_ios` ties up physical devices and, because that pool mixes M1 and M4 Pro hosts, splits every metric into two disconnected series on Skia Perf (device_type=iPhone_11 vs iPhone_16_Pro) with a ~40-60% timing gap between them. Per the review discussion on flutter#186414, instead of moving them to the unpinned `Mac_arm64` pool: * Re-point the shared `mac_benchmark` platform from the Intel `Macmini8,1` (being turned down, flutter.dev/go/macos-intel-deprecation) to the Apple Silicon M1 `Macmini9,1` and add `cpu: arm64`. * Move the four iOS compile benchmarks to `Mac_benchmark` with `bringup: true`. They inherit `device_type: none` and the `["devicelab", "hostonly", "mac", "arm64"]` tags from the platform (dropping the `ios` tag also stops the recipe from running `ideviceinfo`, which would fail on a host-only bot). `flutter_gallery_ios__compile` keeps the `ruby` dependency for CocoaPods, like `flutter_gallery_macos__compile`. `imitation_game_swiftui` stays on `Mac_arm64_ios` because it runs `xcodebuild test` against a physical device. Note that every existing `Mac_benchmark` target moves from Intel to M1 hardware as part of this change, so their benchmark baselines will shift once this lands.
okorohelijah
force-pushed
the
migrate_ios_driver_tests_batch_5
branch
from
September 18, 2026 23:27
336e210 to
11522a0
Compare
vashworth
approved these changes
Sep 23, 2026
This was referenced Sep 23, 2026
7 of 11 tasks
pull Bot
pushed a commit
to AbhiShake1/flutter
that referenced
this pull request
Sep 24, 2026
<!-- Thanks for filing a pull request! Reviewers are typically assigned within a week of filing a request. To learn more about code review, see our documentation on Tree Hygiene: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md --> remove compile benchmarks from bringup Part of flutter#192472 *If you had to change anything in the [flutter/tests] repo, include a link to the migration guide as per the [breaking change policy].* ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant in-code documentation (doc comments with `///`). - [ ] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
This was referenced Sep 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #186414
This batch moves the 4 iOS compile benchmarks off physical-device bots and onto the pinned benchmark pool:
basic_material_app_ios__compileflutter_gallery_ios__compilehello_world_ios__compileimitation_game_flutter(task:imitation_game_flutter__compile)These tasks only run
flutter build iosand never launch an app on the attached iPhone (details in #186414 (comment)). OnMac_arm64_iosthey tie up physical devices and, because that pool mixes M1 and M4 Pro hosts, every metric is split into two disconnected Skia Perf series (device_type=iPhone_11vsiPhone_16_Pro) with a ~40–60% timing gap.Per #186414 (comment), instead of moving them to the unpinned
Mac_arm64pool:Changes
platform_properties.mac_benchmark:mac_model"Macmini8,1"(Intel, being turned down per flutter.dev/go/macos-intel-deprecation) →"Macmini9,1"(Apple Silicon M1 Mac mini), pluscpu: arm64and anarm64tag.Mac_arm64_iostoMac_benchmarkwithbringup: true. They inheritdevice_type: noneand["devicelab", "hostonly", "mac", "arm64"]from the platform; dropping theiostag also stops the devicelab recipe from runningideviceinfo(which would fail on a host-only bot).flutter_gallery_ios__compilekeeps therubydependency for CocoaPods, matchingflutter_gallery_macos__compile.imitation_game_swiftuistays onMac_arm64_ios— it runsxcodebuild testagainst a physical device.Heads-up for the metrics sheriff
Mac_benchmarkmacOS targets (*_macos__compile,*_macos__start_up,complex_layout_scroll_perf_macos*,new_gallery_macos_impeller__transition_perf,flutter_tool_startup__macos, …) move from Intel to M1 hardware with this change, so their baselines will shift (and thearchbenchmark tag changes frominteltoarm). Expect one-time step changes on Skia Perf rather than regressions.Macmini9,1capacity inluci.flutter.prodis ~56 bots (vs ~37 IntelMacmini8,1), shared withMac_arm64; today that pool is all M1, so the pin mostly future-proofs against M4 Pros being added.