Skip to content

Collect non-benchmark tests which use a physical device #142070

Open

Description

@keyonghan

Most non-benchmark tests should be able to run without a physical device, and an emulator should be used instead to save limited DeviceLab resources and potentially improve the test execution time.

This bug tracks to collect the list.

/cc @matanlurey @jonahwilliams @jmagman

Activity

  1. self-assigned this
    on Jan 23, 2024
  2. matanlurey commented on Jan 23, 2024

    @matanlurey
    Contributor

    Thank you! Here is an easy one:

      - name: Linux_pixel_7pro external_ui_integration_test
      - name: Mac_ios external_ui_integration_test_ios

    These don't need a physical device.

  3. jmagman commented on Jan 23, 2024

    @jmagman
    Member

    external_ui_integration_test_ios is using the dev/devicelab/integration_test runners which check which devices are running, and pass -d to the tool, and fail if there are no devices tethered. So this change isn't just to change the bot dimensions or move it to a new pool, it would involve swapping them to a different or more flexible test runner, like moving them into the tools tests (like #73577) or rewriting them to handle simulators like

    await testWithNewIOSSimulator('TestAdd2AppSim', (String deviceId) async {

    Same for:
    channels_integration_test_ios
    route_test_ios
    flavors_test_ios
    flavors_test_ios_xcode_debug
    integration_test_test_ios
    integration_ui_ios_driver
    integration_ui_ios_driver_xcode_debug
    and others on the devicelab/devicelab_drone recipe.

  4. jonahwilliams commented on Jan 23, 2024

    @jonahwilliams
    Contributor

    Some more tests that can be emulators

      - name: Linux_pixel_7pro android_defines_test
      - name: Linux_pixel_7pro android_obfuscate_test
      - name: Linux_pixel_7pro android_semantics_integration_test
      - name: Linux_pixel_7pro basic_material_app_android__compile
      - name: Linux_pixel_7pro channels_integration_test
      - name: Linux_pixel_7pro drive_perf_debug_warning
      - name: Linux_pixel_7pro embedded_android_views_integration_test
      - name: Linux_pixel_7pro external_ui_integration_test
      - name: Linux_pixel_7pro flavors_test
      - name: Linux_pixel_7pro flutter_gallery_android__compile
      - name: Linux_pixel_7pro frame_policy_delay_test_android
      - name: Linux_pixel_7pro integration_ui_driver
      - name: Linux_pixel_7pro integration_ui_keyboard_resize
      - name: Linux_pixel_7pro integration_ui_textfield
      - name: Linux_pixel_7pro native_assets_android
      - name: Linux_pixel_7pro platform_channel_sample_test
      - name: Linux_pixel_7pro platform_interaction_test
      - name: Linux_pixel_7pro routing_test
      - name: Linux_pixel_7pro service_extensions_test
      - name: Linux_pixel_7pro android_choreographer_do_frame_test
      - name: Mac_pixel_7pro entrypoint_dart_registrant
      - name: Mac_pixel_7pro hello_world_android__compile
      - name: Mac_pixel_7pro integration_test_test
      - name: Mac_pixel_7pro integration_ui_frame_number
      - name: Mac_pixel_7pro native_assets_android
      - name: Mac_pixel_7pro run_debug_test_android
      - name: Mac_pixel_7pro run_release_test
    

    Not sure about this one (@yjbanov ) but it looks like this should not be a device test at all

      - name: Linux_pixel_7pro flutter_gallery_v2_chrome_run_test
      - name: Linux_pixel_7pro linux_chrome_dev_mode
    
  5. jmagman commented on Jan 24, 2024

    @jmagman
    Member

    I can work on getting some of the iOS ones to run on a simulator, hopefully not much more complicated than piping through a device override:

    deviceOperatingSystem = DeviceOperatingSystem.ios;
    String? simulatorDeviceId;
    try {
    await testWithNewIOSSimulator('TestHotReloadSim', (String deviceId) async {
    simulatorDeviceId = deviceId;
    // This isn't actually a benchmark test, so do not use the returned `benchmarkScoreKeys` result.
    await createHotModeTest(deviceIdOverride: deviceId, checkAppRunningOnLocalDevice: true)();

  6. keyonghan commented on Jan 24, 2024

    @keyonghan
    ContributorAuthor

    Tl;dr: please confirm the test feasibility to switch to a VM or a VM with an emulator using this sheet: go/flutter_devicelab_to_vm_list.

    Thanks @jonahwilliams @jmagman and @matanlurey for the initial input! I have scanned through all our benchmarks and created the test list combining what you have provided (categorized based on test owners).

    There targets that ones that are NOT uploading metrics to Skia (NOT benchmarks), but are running with a Device.
    Please help identify/confirm those that can be switched to a VM or a VM with an emulator (based on test ownership) using this sheet, and I will update the status here after finalized accordingly). Feel free to loop other stakeholders who may have more context. Thanks!

    • @camsim99
      • Mac_ios spell_check_test
    • @christopherfujino
      • Mac_ios ios_content_validation_test
      • Mac_arm64_ios ios_content_validation_test
    • @dcharkes
      • Linux_pixel_7pro native_assets_android
      • Windows_android native_assets_android
      • Mac_ios native_assets_ios
    • @dnfield
      • Linux_pixel_7pro integration_ui_frame_number
      • Mac_ios integration_ui_ios_frame_number
    • @gaaclarke
      • Linux_pixel_7pro animated_advanced_blend_perf_opengles__timeline_summary
      • Linux_pixel_7pro animated_advanced_blend_perf__timeline_summary
      • Linux_pixel_7pro animated_blur_backdrop_filter_perf_opengles__timeline_summary
      • Linux_pixel_7pro entrypoint_dart_registrant
      • Linux_pixel_7pro hello_world_impeller
      • Mac_ios wide_gamut_ios
    • @HansMuller
    • @hellohuanlin
      • Mac_ios native_platform_view_ui_tests_ios
    • @jonahwilliams
      • Linux_pixel_7pro animated_blur_backdrop_filter_perf__timeline_summary
      • Linux_pixel_7pro draw_vertices_perf_opengles__timeline_summary
      • Linux_pixel_7pro draw_vertices_perf__timeline_summary
    • @reidbaker
      • Linux_pixel_7pro android_choreographer_do_frame_test
    • @stuartmorgan
      • Linux_android hybrid_android_views_integration_test
      • Linux_pixel_7pro embedded_android_views_integration_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro platform_interaction_test
      • Mac_ios ios_defines_test
      • Mac_ios ios_platform_view_tests
      • Mac_ios platform_view_ios__start_up
    • @vashworth
      • Mac_arm64_ios module_test_ios
      • Mac_arm64_ios flavors_test_ios_xcode_debug (confirmed by @jmagman above)
      • Mac_ios channels_integration_test_ios (confirmed by @jmagman above)
      • Mac_ios flavors_test_ios (confirmed by @jmagman above)
      • Mac_ios integration_test_test_ios (confirmed by @jmagman above)
      • Mac_ios integration_ui_ios_driver (confirmed by @jmagman above)
      • Mac_ios integration_ui_ios_driver_xcode_debug (confirmed by @jmagman above)
      • Mac_ios integration_ui_ios_keyboard_resize
      • Mac_ios integration_ui_ios_textfield
      • Mac_ios platform_channel_sample_test_ios
      • Mac_ios platform_channel_sample_test_swift
      • Mac_ios platform_interaction_test_ios
      • Mac_ios route_test_ios
    • @zanderso
      • Linux_android devtools_profile_start_test
      • Linux_android flutter_view__start_up
      • Linux_android platform_view__start_up
      • Linux_pixel_7pro android_defines_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro android_obfuscate_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro basic_material_app_android__compile (confirmed by @jonahwilliams above but is a benchmark test. Please DOUBLE confirm.)
      • Linux_pixel_7pro channels_integration_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro drive_perf_debug_warning (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro external_ui_integration_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro flavors_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro flutter_gallery_android__compile (confirmed by @jonahwilliams above but is a benchmark test. Please DOUBLE confirm.)
      • Linux_pixel_7pro frame_policy_delay_test_android (confirmed by @jonahwilliams above but is a benchmark test. Please DOUBLE confirm.)
      • Linux_pixel_7pro hello_world_android__compile
      • Linux_pixel_7pro imagefiltered_transform_animation_perf__timeline_summary
      • Linux_pixel_7pro integration_test_test
      • Linux_pixel_7pro integration_ui_driver (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro integration_ui_keyboard_resize (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro integration_ui_textfield
      • Linux_pixel_7pro linux_chrome_dev_mode (mentioned by @jonahwilliams above but is a benchmark test. Please DOUBLE confirm.)
      • Linux_pixel_7pro platform_channel_sample_test
      • Linux_pixel_7pro routing_test
      • Linux_pixel_7pro run_debug_test_android
      • Linux_pixel_7pro run_release_test (confirmed by @jonahwilliams above)
      • Linux_pixel_7pro service_extensions_test
      • Mac_ios external_ui_integration_test_ios (confirmed by @matanlurey above)
      • Mac_ios flutter_view_ios__start_up
      • Windows_android channels_integration_test_win
      • Windows_android flavors_test_win
    • @yjbanov
      • Linux_pixel_7pro flutter_gallery_v2_chrome_run_test
  7. jonahwilliams commented on Jan 24, 2024

    @jonahwilliams
    Contributor
    Linux_pixel_7pro animated_blur_backdrop_filter_perf__timeline_summary
    Linux_pixel_7pro draw_vertices_perf_opengles__timeline_summary
    Linux_pixel_7pro draw_vertices_perf__timeline_summary
    

    These are definitely benchmarks.

    basic_material_app_android__compile
    

    This is a benchmark but it is reporting the compile times from the host, so the target device is irrelevant.

    linux_chrome_dev_mode
    

    This looks like a web benchmark that is running on the host. @yjbanov would need to confirm.

    frame_policy_delay_test_android
    

    ACK, this one is actually a benchmark, strike it from the list.

  8. dcharkes commented on Jan 24, 2024

    @dcharkes
    Contributor

    @dcharkes

    • Linux_pixel_7pro native_assets_android
    • Windows_android native_assets_android
    • Mac_ios native_assets_ios

    At least for ios the simulator needs less signing than the physical device, so to keep test coverage for the signing, we should keep running that one on device.

    For the Android ones I'd be a little uncomfortable if we don't test on any physical device. There might be some differences between emulator and physical device, but nothing pops to mind. cc @reidbaker @stuartmorgan

  9. jonahwilliams commented on Jan 24, 2024

    @jonahwilliams
    Contributor

    The tool literally cannot tell the difference between Android emulators/simulators, beyond architecture (in fact, a number of flagship phones claim to be emulators in their adb config). As long as you can get an arm/arm64 emulator, there should not be any differences.

  10. dcharkes commented on Jan 24, 2024

    @dcharkes
    Contributor

    We did have this weird behavior difference between the emulator and physical device:

    But if we're struggling for capacity maybe we don't care about the minimal differences between device and emulator. @reidbaker Should probably make the call here.

  11. reidbaker commented on Jan 24, 2024

    @reidbaker
    Member

    At a high level I think that we should be able to run on real devices provided the tests are in post submit and real devices long-term are important for us to test on.

    I hear the arguments for why emulators are better, especially in presubmit where we have the ability to scale.

  12. added
    P2Important issues not at the top of the work list
    triaged-infraTriaged by Infrastructure team
    on Jan 25, 2024
  13. jmagman commented on Jan 25, 2024

    @jmagman
    Member

    Here's an example of a test moving out of devicelab runner into the tool integration tests: #70978 and #74685

  14. jmagman commented on Jan 25, 2024

    @jmagman
    Member

    Here's an example of a test moving out of devicelab runner into the tool integration tests: #70978 and #74685

    To expand on this, some of the tests running in the devicelab or on devicelab recipe aren't doing anything on a target device, either physical or emulator/simulator. They are building or otherwise validating apps on only a host machine.

    The runProjectTest / runPluginProjectTest / runModuleProjectTest tests are good candidates to be moved into the tool integration tests.
    android_obfuscate_test (currently using a Linux/Pixel devicelab bot)
    engine_dependency_proxy_test (already running in a VM)
    gradle_desugar_classes_test (VM)
    gradle_java8_compile_test (VM)
    gradle_plugin_bundle_test (VM)
    gradle_plugin_fat_apk_test (VM)
    gradle_plugin_light_apk_test (VM)
    ios_content_validation_test (currently using a Mac/iOS devicelab bot) PR to move: #142272

  15. keyonghan commented on Jan 30, 2024

    @keyonghan
    ContributorAuthor
    basic_material_app_android__compile
    

    This is a benchmark but it is reporting the compile times from the host, so the target device is irrelevant.

    Sounds a good fit for a host only benchmark (may not be suitable for a VM): #126090

  16. jmagman commented on Jan 30, 2024

    @jmagman
    Member

    Sounds a good fit for a host only benchmark (may not be suitable for a VM): #126090

    #142280 (comment)

    hot_mode_dev_cycle is the only test I'm aware of where we need a pinned physical device and pinned host hardware since the benchmark is measuring hot reload times, so time for the host to prep it and the device to apply it.

    The __compile ones are hostonly benchmarks, they are gathering metrics for how long compilation/subsequent complication takes. So they do not run on physical mobile devices, but should run on physical hosts pinned to consistent hardware. Like I don't know why we have a Mac_pixel_7pro hello_world_android__compile and Mac_arm64_android hello_world_android__compile tests, they are doing the same and should be on Mac_benchmark.

  17. flutter-triage-bot commented on Jun 4, 2024

    @flutter-triage-bot

    This issue is assigned to @keyonghan but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

  18. added
    ⏳Bot is counting down the days until it unassigns the issue
    on Jun 4, 2024
  19. removed
    ⏳Bot is counting down the days until it unassigns the issue
    on Jun 5, 2024
  20. matanlurey commented on Sep 30, 2024

    @matanlurey
    Contributor

    @jmagman chatted with me offline in a group chat, and mentioned two bits for aspiring migrators:

    With our existing runners it's a lot of work to move out of dev/devicelab/bin/tasks/, the test essentially needs to be rewritten. Here's an example where I moved one from that runner into the tools integration test runner #74685

    A more scalable way to do this would be to decouple dev/devicelab/ from the actual devicelab and instead allow that runner to target a simulator, instead of the currently tethered physical device

    /cc @jtmcdole

  21. jmagman commented on Sep 30, 2024

    @jmagman
    Member

    A more scalable way to do this would be to decouple dev/devicelab/ from the actual devicelab and instead allow that runner to target a simulator, instead of the currently tethered physical device

    (Similar to this #142070 (comment))

  22. self-assigned this
    on Feb 22, 2025
  23. removed their assignment
    on Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions