Skip to content

Decide on an Android device testing policy for flutter/packages #131429

@stuartmorgan-g

Description

@stuartmorgan-g

Historically we did not have emulator support for tests for Android in flutter/packages, so all Android integration testing was done on device via FTL. Now that we have emulator tests (other than camera; see #130987) it's not clear what the physical device testing strategy should be:

  • Eliminate them completely?
  • Make them post-submit only?
  • Keep the duplicate testing everywhere?
  • Pare it down to a limit set of plugins and/or tests?

In the short term, since we don't have any data about the relative ability of integration tests vs device tests to catch issues for us in practice, we're going to keep duplicate tests. The main android_platform_tests tasks will run emulator (along with all the other standard platform tests, like build and native unit), and a new set of tasks will be brought up to run just the FTL tests.

After we have a few months of data comparing its effectiveness to emulator tests, we should revisit if/how we want to continue the use of FTL. (In the meantime, this does mean we'll double our exposure to integration test flake for Android, and integration tests are—across platforms—by far the flakiest of tests.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.team-ecosystemOwned by Ecosystem teamtriaged-ecosystemTriaged by Ecosystem team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions