You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Collect non-benchmark tests which use a physical device #142070
Most non-benchmark tests should be able to run without a physical device, and an emulator should be used instead to save limited DeviceLab resources and potentially improve the test execution time.
external_ui_integration_test_ios is using the dev/devicelab/integration_test runners which check which devices are running, and pass -d to the tool, and fail if there are no devices tethered. So this change isn't just to change the bot dimensions or move it to a new pool, it would involve swapping them to a different or more flexible test runner, like moving them into the tools tests (like #73577) or rewriting them to handle simulators like
Same for: channels_integration_test_ios route_test_ios flavors_test_ios flavors_test_ios_xcode_debug integration_test_test_ios integration_ui_ios_driver integration_ui_ios_driver_xcode_debug
and others on the devicelab/devicelab_drone recipe.
Tl;dr: please confirm the test feasibility to switch to a VM or a VM with an emulator using this sheet: go/flutter_devicelab_to_vm_list.
Thanks @jonahwilliams@jmagman and @matanlurey for the initial input! I have scanned through all our benchmarks and created the test list combining what you have provided (categorized based on test owners).
There targets that ones that are NOT uploading metrics to Skia (NOT benchmarks), but are running with a Device.
Please help identify/confirm those that can be switched to a VM or a VM with an emulator (based on test ownership) using this sheet, and I will update the status here after finalized accordingly). Feel free to loop other stakeholders who may have more context. Thanks!
At least for ios the simulator needs less signing than the physical device, so to keep test coverage for the signing, we should keep running that one on device.
For the Android ones I'd be a little uncomfortable if we don't test on any physical device. There might be some differences between emulator and physical device, but nothing pops to mind. cc @reidbaker@stuartmorgan
The tool literally cannot tell the difference between Android emulators/simulators, beyond architecture (in fact, a number of flagship phones claim to be emulators in their adb config). As long as you can get an arm/arm64 emulator, there should not be any differences.
But if we're struggling for capacity maybe we don't care about the minimal differences between device and emulator. @reidbaker Should probably make the call here.
At a high level I think that we should be able to run on real devices provided the tests are in post submit and real devices long-term are important for us to test on.
I hear the arguments for why emulators are better, especially in presubmit where we have the ability to scale.
Here's an example of a test moving out of devicelab runner into the tool integration tests: #70978 and #74685
To expand on this, some of the tests running in the devicelab or on devicelab recipe aren't doing anything on a target device, either physical or emulator/simulator. They are building or otherwise validating apps on only a host machine.
The runProjectTest / runPluginProjectTest / runModuleProjectTest tests are good candidates to be moved into the tool integration tests.
android_obfuscate_test (currently using a Linux/Pixel devicelab bot)
engine_dependency_proxy_test (already running in a VM)
gradle_desugar_classes_test (VM)
gradle_java8_compile_test (VM)
gradle_plugin_bundle_test (VM)
gradle_plugin_fat_apk_test (VM)
gradle_plugin_light_apk_test (VM)
ios_content_validation_test (currently using a Mac/iOS devicelab bot) PR to move: #142272
hot_mode_dev_cycle is the only test I'm aware of where we need a pinned physical device and pinned host hardware since the benchmark is measuring hot reload times, so time for the host to prep it and the device to apply it.
The __compile ones are hostonly benchmarks, they are gathering metrics for how long compilation/subsequent complication takes. So they do not run on physical mobile devices, but should run on physical hosts pinned to consistent hardware. Like I don't know why we have a Mac_pixel_7pro hello_world_android__compile and Mac_arm64_android hello_world_android__compile tests, they are doing the same and should be on Mac_benchmark.
This issue is assigned to @keyonghan but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!
@jmagman chatted with me offline in a group chat, and mentioned two bits for aspiring migrators:
With our existing runners it's a lot of work to move out of dev/devicelab/bin/tasks/, the test essentially needs to be rewritten. Here's an example where I moved one from that runner into the tools integration test runner #74685
A more scalable way to do this would be to decouple dev/devicelab/ from the actual devicelab and instead allow that runner to target a simulator, instead of the currently tethered physical device
A more scalable way to do this would be to decouple dev/devicelab/ from the actual devicelab and instead allow that runner to target a simulator, instead of the currently tethered physical device
Activity
matanlurey commentedon Jan 23, 2024
Thank you! Here is an easy one:
These don't need a physical device.
jmagman commentedon Jan 23, 2024
external_ui_integration_test_iosis using thedev/devicelab/integration_testrunners which check which devices are running, and pass-dto the tool, and fail if there are no devices tethered. So this change isn't just to change the bot dimensions or move it to a new pool, it would involve swapping them to a different or more flexible test runner, like moving them into the tools tests (like #73577) or rewriting them to handle simulators likeflutter/dev/devicelab/bin/tasks/module_test_ios.dart
Line 549 in cbe0cea
Same for:
channels_integration_test_iosroute_test_iosflavors_test_iosflavors_test_ios_xcode_debugintegration_test_test_iosintegration_ui_ios_driverintegration_ui_ios_driver_xcode_debugand others on the
devicelab/devicelab_dronerecipe.jonahwilliams commentedon Jan 23, 2024
Some more tests that can be emulators
Not sure about this one (@yjbanov ) but it looks like this should not be a device test at all
jmagman commentedon Jan 24, 2024
I can work on getting some of the iOS ones to run on a simulator, hopefully not much more complicated than piping through a device override:
flutter/dev/devicelab/bin/tasks/hot_mode_dev_cycle_ios_simulator.dart
Lines 13 to 19 in 90e4a12
keyonghan commentedon Jan 24, 2024
Tl;dr: please confirm the test feasibility to switch to a VM or a VM with an emulator using this sheet: go/flutter_devicelab_to_vm_list.
Thanks @jonahwilliams @jmagman and @matanlurey for the initial input! I have scanned through all our benchmarks and created the test list combining what you have provided (categorized based on test owners).
There targets that ones that are NOT uploading metrics to Skia (NOT benchmarks), but are running with a Device.
Please help identify/confirm those that can be switched to a VM or a VM with an emulator (based on test ownership) using this sheet, and I will update the status here after finalized accordingly). Feel free to loop other stakeholders who may have more context. Thanks!
jonahwilliams commentedon Jan 24, 2024
These are definitely benchmarks.
This is a benchmark but it is reporting the compile times from the host, so the target device is irrelevant.
This looks like a web benchmark that is running on the host. @yjbanov would need to confirm.
ACK, this one is actually a benchmark, strike it from the list.
dcharkes commentedon Jan 24, 2024
At least for ios the simulator needs less signing than the physical device, so to keep test coverage for the signing, we should keep running that one on device.
For the Android ones I'd be a little uncomfortable if we don't test on any physical device. There might be some differences between emulator and physical device, but nothing pops to mind. cc @reidbaker @stuartmorgan
jonahwilliams commentedon Jan 24, 2024
The tool literally cannot tell the difference between Android emulators/simulators, beyond architecture (in fact, a number of flagship phones claim to be emulators in their adb config). As long as you can get an arm/arm64 emulator, there should not be any differences.
dcharkes commentedon Jan 24, 2024
We did have this weird behavior difference between the emulator and physical device:
DynamicLibrary.process()on Android emuators dart-lang/sdk#53249But if we're struggling for capacity maybe we don't care about the minimal differences between device and emulator. @reidbaker Should probably make the call here.
reidbaker commentedon Jan 24, 2024
At a high level I think that we should be able to run on real devices provided the tests are in post submit and real devices long-term are important for us to test on.
I hear the arguments for why emulators are better, especially in presubmit where we have the ability to scale.
jmagman commentedon Jan 25, 2024
Here's an example of a test moving out of devicelab runner into the tool integration tests: #70978 and #74685
jmagman commentedon Jan 25, 2024
To expand on this, some of the tests running in the devicelab or on devicelab recipe aren't doing anything on a target device, either physical or emulator/simulator. They are building or otherwise validating apps on only a host machine.
The
runProjectTest/runPluginProjectTest/runModuleProjectTesttests are good candidates to be moved into the tool integration tests.android_obfuscate_test (currently using a Linux/Pixel devicelab bot)
engine_dependency_proxy_test (already running in a VM)
gradle_desugar_classes_test (VM)
gradle_java8_compile_test (VM)
gradle_plugin_bundle_test (VM)
gradle_plugin_fat_apk_test (VM)
gradle_plugin_light_apk_test (VM)
ios_content_validation_test (currently using a Mac/iOS devicelab bot) PR to move: #142272
Move iOS content validation devicelab test into tool integration test (…
keyonghan commentedon Jan 30, 2024
Sounds a good fit for a host only benchmark (may not be suitable for a VM): #126090
jmagman commentedon Jan 30, 2024
#142280 (comment)
flutter-triage-bot commentedon Jun 4, 2024
This issue is assigned to @keyonghan but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!
matanlurey commentedon Sep 30, 2024
@jmagman chatted with me offline in a group chat, and mentioned two bits for aspiring migrators:
/cc @jtmcdole
jmagman commentedon Sep 30, 2024
(Similar to this #142070 (comment))