[iOS] Migrate non-benchmark devicelab tests that don't need a physical device

iOS specific version of #142070.

If a test is using platform properties Mac_arm64_ios, Mac_x64_ios, or Mac_ios, that means it's using a physical device.

We should audit each of these and determine if they need to use a device.

If it doesn't need a device, we should make it host-only (chromium bot). We should also consider moving to the integration_shard (see [flutter_tools] Migrate hostonly_devicelab tests to flutter_tools #65790).
If it needs a device, but it's not a benchmark or deployment workflow test, we should migrate it to use a simulator and a host-only bot (chromium bot)

Proposal

This is a prototype PR to propose and discuss the architectural strategy for moving iOS UI/Functional tests off physical lab devices and onto Mac VM simulators (resolving #186414) before applying the pattern broadly.

The Approach:
Instead of ripping these UI tests out of the dev/devicelab harness and rewriting them into the test.dart integration shards, we are keeping them within the Devicelab framework. We decouple the tests from physical hardware by injecting an isolated, ephemeral iOS Simulator during the test's execution phase, allowing us to safely migrate the .ci.yaml targets from the physical hardware pool (Mac_ios) to the standard Swarming VM pool (Mac_arm64).

Why keep them in Devicelab instead of rewriting them?

Preservation of Testing Frameworks: The devicelab tasks heavily utilize the IntegrationTest and DriverTest factory classes, which handle complex, test-specific orchestration (temporary project scaffolding, --dart-define injections, platform setups). Moving to test.dart would require discarding and recreating all this framework glue.
CI Isolation: Devicelab maps one .ci.yaml builder strictly to one test script. When a test crashes or hangs, you immediately get a 1:1 mapped LUCI build failure. Integration shards bunch many tests together, meaning one severe crash/hang can take down the whole shard and mask downstream failures.
Safety & Reversibility: If a specific test ends up catching an obscure bug that only manifests on physical iOS devices, we can instantly revert that specific test back to the Mac_ios hardware pool without needing to rewrite it back into the Devicelab framework.

Trade-offs:

Moving away from a globally provisioned simulator (like in test.dart shards) means we incur a small overhead to manage the simulator boot/teardown lifecycle within each isolated task script.

Here is the comprehensive list of Devicelab targets I plan to migrate using this strategy.

Compile-Only Tasks

These tasks only verify that an iOS project can successfully build (running flutter build ios and inspecting the generated artifacts). Because they never invoke devices.workingDevice or launch an app, they can be safely moved to the Mac VM pool with a simple .ci.yaml change. (Note: Tests like build_ios_framework_module_test were already running on Mac_arm64 and are thus omitted).

basic_material_app_ios__compile (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472)
flutter_gallery_ios__compile (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472)
hello_world_ios__compile (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472)
imitation_game_flutter__compile (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472)
imitation_game_swiftui__compile (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472)

Functional / UI Tests

These tasks verify UI layout, platform channels, and debugging workflows. While they do require launching an app, they do not measure raw device performance (unlike our benchmarking tasks, which must remain on physical hardware to prevent skewed metrics). We will migrate these by injecting the testWithNewIOSSimulator wrapper.

@okorohelijah

@okorohelijah native_assets_ios and ios_debug_workflow are intended to run on a physical device

Here are some examples where we moved devicelab directory tests into the tool integration.shard:
#73577
#80161
Those are both build-only tests, but we do have examples of testing on simulators in the tool shard (I believe).

And also be careful moving around the *__compile tests. They are benchmarks, so would be noisy (and maybe are now) if they are run on different Mac hardware configurations (slow vs fast machine). You may need to pin them on a particular mac_model so they run consistently.

flutter/.ci.yaml

Line 207 in 24debf8

mac_model: "Macmini8,1"

I updated the list to remove platform_channel_sample_test_ios and platform_channel_sample_test_swift. I checked their native code and they both invoke UIDevice.currentDevice.batteryLevel. On an iOS Simulator, this API returns an unknown state (-1), which causes the Dart test driver to throw a "Failed to get battery level" error instead of the expected percentage.

Migrate 5 iOS UI driver tests to Mac Simulators (flutter#190679)

I updated the list to remove platform_channel_sample_test_ios and platform_channel_sample_test_swift. I checked their native code and they both invoke UIDevice.currentDevice.batteryLevel. On an iOS Simulator, this API returns an unknown state (-1), which causes the Dart test driver to throw a "Failed to get battery level" error instead of the expected percentage.

The batteryLevel example is pretty arbitrary for testing platform channels, you could migrate the test to something that works consistently on both simulator and physical.

Migrate ios driver tests batch 3 to simulators (flutter#191333)

@jmagman

[@jmagman ](https://github.com/jmagman) Thanks for calling out that the *__compile tasks are actually recording benchmarks. Moving them to an unpinned Mac_arm64 VM pool would definitely introduce noise to the performance metrics. While moving them to the tool tests (like PR #73577) is an option, our primary goal here is to free up the physical device pool. I think the safest action is to just leave all compile benchmarks on the physical Mac_ios hardware to preserve their metrics integrity.

I don't think we should skip the compile tests just to not disrupt the benchmarks. We just need to make sure that they use specific mac_model as @jmagman mentioned. This is an opportunity to improve them. Those tests are on Mac_arm64_ios, which does not pin the mac_model so are likely already super noisy since some of the bots are M4s and some are M1s. Perf appears to be down right now so I can't check the actual benchmarks, but if you look at the builds time https://ci.chromium.org/ui/p/flutter/builders/luci.flutter.prod/Mac_arm64_ios%20basic_material_app_ios__compile, you can see they already have a discrepancy per bot hardware.

One possible solution is to use Mac_benchmark, although that will pin them to Intel devices

Migrate ios driver tests batch 2 to simulators (flutter#190930)

@vashworth

Following up on the discussion regarding the compile benchmarks (basic_material_app_ios__compile, flutter_gallery_ios__compile, hello_world_ios__compile, and imitation_game_flutter):

I investigated the current Skia Perf graphs and telemetry for these tasks, and here is what is happening:

1. Current State & Telemetry Findings

Fractured Metrics: Because these tasks currently run on Mac_arm64_ios, metrics_center.dart grabs the attached physical phone model. Every single metric is split into two disconnected series on Skia Perf (device_type=iPhone_11 vs device_type=iPhone_16_Pro).
Hardware Variance: The testbeds with iPhone 16 Pros attached (M4 Macs) compile ~40%–60% faster than the testbeds with iPhone 11s attached (M1 Macs) (e.g., ~27s vs. ~45s release compile time for basic_material_app). This massive swing drowns out any actual compiler or tooling regressions.
Unnecessary Phone Usage: These tasks only execute flutter build ios—they never launch or interact with the attached device (the output binary sizes are 100% identical). Running them on Mac_arm64_ios unnecessarily ties up scarce physical iPhones in the lab.

2. Tradeoffs of `Mac_benchmark` vs. Intel Deprecation

While moving them to Mac_benchmark would pin them to a consistent hardware model (Macmini8,1), it runs into a major conflict:

We are actively deprecating and turning down Intel Mac bots across CI (per flutter.dev/go/macos-intel-deprecation and Remove Intel test builders that already run on ARM #189295 / Stop running most Mac x64 builders that have Mac ARM equivalents on master #189301).
Adding new workloads to Intel Macmini8,1 increases reliance on aging hardware that is slated for decommissioning, creating technical debt we will have to migrate again shortly.

3. Proposed Plan for Batch 5

Following the pattern from #189301 (which migrated hot_mode_dev_cycle_macos_target__benchmark off Intel onto Mac_arm64), I propose:

Migrate the iOS compile benchmarks to Mac_arm64 hostonly:
- basic_material_app_ios__compile
- flutter_gallery_ios__compile
- hello_world_ios__compile
- imitation_game_flutter (task: imitation_game_flutter__compile)
- Tag them with ["devicelab", "hostonly", "mac", "arm64"] and device_type: none.
- Benefits: Immediately frees up physical iPhones, removes the artificial iPhone_11 vs iPhone_16_Pro line split in Skia Perf, and avoids taking on new Intel hardware dependencies.
Keep imitation_game_swiftui on Mac_arm64_ios:
- Unlike the compile tasks, imitation_game_swiftui executes xcodebuild test directly against a real device to measure time_to_first_frame, so it must remain on physical hardware.
Future Follow-up:
- Once infra establishes a dedicated Apple Silicon benchmark pool with pinned mac_model (e.g., Mac_arm64_benchmark), these and existing Mac_arm64 benchmarks can be pinned there to resolve the M1 vs. M4 host variance.

@vashworth @jmagman wdyt?

Migrate ios driver tests batch 4 to simulators (flutter#192463)

Let's go ahead and skip to 3:

Future Follow-up:
Once infra establishes a dedicated Apple Silicon benchmark pool with pinned mac_model (e.g., Mac_arm64_benchmark), these and existing Mac_arm64 benchmarks can be pinned there to resolve the M1 vs. M4 host variance.

It's not up the Infra team to establish that, it's our team's domain. ~~Let's go ahead and introduce Mac_arm64_benchmark and pin it to M1s.~~

Actually, even better would be to change Mac benchmark's mac_model to the arm M1 hardware since we want to stop testing Intel anyways.

Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1

Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to…

@jmagman

Wrapping this one up. I went back through .ci.yaml at HEAD and every iOS target still on Mac_ios / Mac_arm64_ios / Mac_x64_ios is now either a benchmark or a test that genuinely needs the hardware.

What moved off physical devices

Simulator on host-only Mac bots (via testWithNewIOSSimulator):

flavors_test_ios (Migrate flavors_test_ios to Simulators and add UIScene support #189442)
integration_ui_ios_driver, integration_ui_ios_frame_number, integration_ui_ios_keyboard_resize, integration_ui_ios_textfield (Migrate 5 iOS UI driver tests to Mac Simulators #190679)
integration_test_test_ios, ios_defines_test, keyboard_hot_restart_ios, native_platform_view_ui_tests_ios, platform_channel_sample_test_ios, platform_channel_sample_test_swift (Migrate ios driver tests batch 3 to simulators #191333). For the two platform channel samples I took @jmagman's suggestion and made the battery example return a fixed value under TARGET_OS_SIMULATOR instead of failing.
channels_integration_test_ios, platform_interaction_test_ios, route_test_ios (Migrate ios driver tests batch 2 to simulators #190930). ios_platform_view_tests was deleted in the same PR since the engine's thread merging made it redundant with native_platform_view_ui_tests_ios.
external_textures_integration_test_ios, hello_world_impeller_ios_sdfs (Migrate ios driver tests batch 4 to simulators #192463)

Host-only, no device at all:

basic_material_app_ios__compile, flutter_gallery_ios__compile, hello_world_ios__compile, imitation_game_flutter moved to Mac_benchmark, and Mac_benchmark itself is now pinned to Macmini9,1 (M1) with cpu: arm64 instead of the Intel Macmini8,1 (Move iOS compile benchmarks to Mac_benchmark and pin Mac_benchmark to M1 #192472, out of bringup in remove compile benchmarks from bringup #193249). That also fixed the split iPhone_11 / iPhone_16_Pro series on Skia Perf.

Two items from the original list turned out not to exist anymore: integration_ui_ios_screenshot was removed back in #115848, and ios_debug_workflow was removed in #191141 once the lldb/devicectl path became the default.

What stays on physical devices, and why

All the *_perf_ios*, *__timeline_summary, *__e2e_summary, *__start_up, *__transition_perf, microbenchmarks_ios, platform_channels_benchmarks_ios, large_image_changer_perf_ios, simple_animation_perf_ios: benchmarks (PerfTest / StartupTest / flutter run --profile), and they run in profile mode, which the simulator doesn't support anyway.
hot_mode_dev_cycle_ios__benchmark: also a benchmark, but it runs flutter run --hot in debug mode (hot reload needs the JIT), so it could run on a simulator — and it already does, as Mac_arm64 hot_mode_dev_cycle_ios_simulator. The device variant stays because its numbers are the ones we track.
flutter_view_ios__start_up, platform_view_ios__start_up: don't upload metrics, but they're profile-mode launches through StartupTest, so they can't run on a simulator either.
imitation_game_swiftui: runs xcodebuild test against the attached phone to measure time to first frame.
native_assets_ios: deployment workflow test, intended for a real device (there's a separate native_assets_ios_simulator already).
wide_gamut_ios: checks Display P3 rendering, which isn't reliable on headless simulator hosts.
Mac_x64_ios integration_test_test_ios and Mac_x64_ios hot_mode_dev_cycle_ios__benchmark only run on beta/stable and are already tagged for removal under Remove Intel test builders that already run on ARM #189295.
spell_check_test_ios: was in the batch 3 draft but we pulled it out of Migrate ios driver tests batch 3 to simulators #191333 (Migrate ios driver tests batch 3 to simulators #191333 (comment)) since simulator text input isn't reliable enough for a spell-check test ([iOS] Tapping "Passwords" closes and reopens the keyboard #190240). Staying on a device.

Closing this out. The generic cross-platform tracking stays in #142070, and the "move devicelab tests into the tool integration shard" idea lives in #65790.

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

Navigation Menu

Repository navigation

[iOS] Migrate non-benchmark devicelab tests that don't need a physical device #186414

Proposal

Compile-Only Tasks

Functional / UI Tests

3 remaining items

1. Current State & Telemetry Findings

2. Tradeoffs of `Mac_benchmark` vs. Intel Deprecation

3. Proposed Plan for Batch 5

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[iOS] Migrate non-benchmark devicelab tests that don't need a physical device #186414

Description

Activity

okorohelijah commented on Jul 30, 2026

Proposal

Compile-Only Tasks

Functional / UI Tests

vashworth commented on Jul 30, 2026

jmagman commented on Jul 31, 2026

jmagman commented on Jul 31, 2026

3 remaining items

okorohelijah commented on Aug 11, 2026

jmagman commented on Aug 14, 2026

vashworth commented on Aug 26, 2026

okorohelijah commented on Sep 9, 2026

1. Current State & Telemetry Findings

2. Tradeoffs of Mac_benchmark vs. Intel Deprecation

3. Proposed Plan for Batch 5

vashworth commented on Sep 18, 2026

okorohelijah commented on Oct 1, 2026

github-actions commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

2. Tradeoffs of `Mac_benchmark` vs. Intel Deprecation