Skip to content

Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 - #192472

Merged
auto-submit[bot] merged 1 commit into
flutter:masterfrom
okorohelijah:migrate_ios_driver_tests_batch_5
Sep 23, 2026
Merged

auto-submit[bot] merged 1 commit into
flutter:masterfrom
okorohelijah:migrate_ios_driver_tests_batch_5

Conversation

@okorohelijah

@okorohelijah okorohelijah commented Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Part of #186414

This batch moves the 4 iOS compile benchmarks off physical-device bots and onto the pinned benchmark pool:

  • basic_material_app_ios__compile
  • flutter_gallery_ios__compile
  • hello_world_ios__compile
  • imitation_game_flutter (task: imitation_game_flutter__compile)

These tasks only run flutter build ios and never launch an app on the attached iPhone (details in #186414 (comment)). On Mac_arm64_ios they tie up physical devices and, because that pool mixes M1 and M4 Pro hosts, every metric is split into two disconnected Skia Perf series (device_type=iPhone_11 vs iPhone_16_Pro) with a ~40–60% timing gap.

Per #186414 (comment), instead of moving them to the unpinned Mac_arm64 pool:

Changes

  • platform_properties.mac_benchmark: mac_model "Macmini8,1" (Intel, being turned down per flutter.dev/go/macos-intel-deprecation) → "Macmini9,1" (Apple Silicon M1 Mac mini), plus cpu: arm64 and an arm64 tag.
  • Moved the 4 targets from Mac_arm64_ios to Mac_benchmark with bringup: true. They inherit device_type: none and ["devicelab", "hostonly", "mac", "arm64"] from the platform; dropping the ios tag also stops the devicelab recipe from running ideviceinfo (which would fail on a host-only bot). flutter_gallery_ios__compile keeps the ruby dependency for CocoaPods, matching flutter_gallery_macos__compile.
  • imitation_game_swiftui stays on Mac_arm64_ios — it runs xcodebuild test against a physical device.

Heads-up for the metrics sheriff

  • All 13 existing Mac_benchmark macOS targets (*_macos__compile, *_macos__start_up, complex_layout_scroll_perf_macos*, new_gallery_macos_impeller__transition_perf, flutter_tool_startup__macos, …) move from Intel to M1 hardware with this change, so their baselines will shift (and the arch benchmark tag changes from intel to arm). Expect one-time step changes on Skia Perf rather than regressions.
  • Host-only Macmini9,1 capacity in luci.flutter.prod is ~56 bots (vs ~37 Intel Macmini8,1), shared with Mac_arm64; today that pool is all M1, so the pin mostly future-proofs against M4 Pros being added.

@okorohelijah
okorohelijah marked this pull request as ready for review September 9, 2026 10:11

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request modifies .ci.yaml by renaming several macOS ARM64 targets to remove the _ios suffix, updating their tags to include hostonly and arm64 instead of ios, and adding a Ruby dependency to the flutter_gallery_ios__compile target. There are no review comments, and I have no feedback to provide.

@Piinks Piinks added platform-ios iOS applications specifically team-ios Owned by iOS platform team labels Sep 9, 2026
@okorohelijah okorohelijah added the CICD Run CI/CD label Sep 18, 2026
Part of flutter#186414

The iOS compile benchmarks (basic_material_app_ios__compile,
flutter_gallery_ios__compile, hello_world_ios__compile and
imitation_game_flutter__compile) only run `flutter build ios`; they never
launch an app on the attached iPhone. Running them on `Mac_arm64_ios`
ties up physical devices and, because that pool mixes M1 and M4 Pro
hosts, splits every metric into two disconnected series on Skia Perf
(device_type=iPhone_11 vs iPhone_16_Pro) with a ~40-60% timing gap
between them.

Per the review discussion on flutter#186414, instead of moving them to the
unpinned `Mac_arm64` pool:

* Re-point the shared `mac_benchmark` platform from the Intel
  `Macmini8,1` (being turned down, flutter.dev/go/macos-intel-deprecation)
  to the Apple Silicon M1 `Macmini9,1` and add `cpu: arm64`.
* Move the four iOS compile benchmarks to `Mac_benchmark` with
  `bringup: true`. They inherit `device_type: none` and the
  `["devicelab", "hostonly", "mac", "arm64"]` tags from the platform
  (dropping the `ios` tag also stops the recipe from running
  `ideviceinfo`, which would fail on a host-only bot).
  `flutter_gallery_ios__compile` keeps the `ruby` dependency for
  CocoaPods, like `flutter_gallery_macos__compile`.

`imitation_game_swiftui` stays on `Mac_arm64_ios` because it runs
`xcodebuild test` against a physical device.

Note that every existing `Mac_benchmark` target moves from Intel to M1
hardware as part of this change, so their benchmark baselines will
shift once this lands.
@okorohelijah
okorohelijah force-pushed the migrate_ios_driver_tests_batch_5 branch from 336e210 to 11522a0 Compare September 18, 2026 23:27
@okorohelijah okorohelijah changed the title Migrate iOS compile benchmarks batch 5 to hostonly Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 Sep 18, 2026
@github-actions github-actions Bot removed platform-ios iOS applications specifically team-ios Owned by iOS platform team labels Sep 18, 2026
@Piinks Piinks added platform-ios iOS applications specifically team-ios Owned by iOS platform team labels Sep 21, 2026
@okorohelijah okorohelijah added the autosubmit Merge PR when tree becomes green via auto submit App label Sep 23, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Sep 23, 2026
Merged via the queue into flutter:master with commit 07510ad Sep 23, 2026
35 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Sep 23, 2026
pull Bot pushed a commit to AbhiShake1/flutter that referenced this pull request Sep 24, 2026
<!--
Thanks for filing a pull request!
Reviewers are typically assigned within a week of filing a request.
To learn more about code review, see our documentation on Tree Hygiene:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
-->

remove compile benchmarks from bringup

Part of flutter#192472

*If you had to change anything in the [flutter/tests] repo, include a
link to the migration guide as per the [breaking change policy].*

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [ ] I updated/added relevant in-code documentation (doc comments with
`///`).
- [ ] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [ ] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [ ] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
@vashworth vashworth added the cp: beta cherry pick this pull request to beta release candidate branch label Sep 30, 2026
@eyebrowsoffire eyebrowsoffire added the cp: stable cherry pick this pull request to stable release candidate branch label Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD cp: beta cherry pick this pull request to beta release candidate branch cp: stable cherry pick this pull request to stable release candidate branch platform-ios iOS applications specifically team-ios Owned by iOS platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants