-
Notifications
You must be signed in to change notification settings - Fork 29.7k
Description
Historically we did not have emulator support for tests for Android in flutter/packages, so all Android integration testing was done on device via FTL. Now that we have emulator tests (other than camera; see #130987) it's not clear what the physical device testing strategy should be:
- Eliminate them completely?
- Make them post-submit only?
- Keep the duplicate testing everywhere?
- Pare it down to a limit set of plugins and/or tests?
In the short term, since we don't have any data about the relative ability of integration tests vs device tests to catch issues for us in practice, we're going to keep duplicate tests. The main android_platform_tests tasks will run emulator (along with all the other standard platform tests, like build and native unit), and a new set of tasks will be brought up to run just the FTL tests.
After we have a few months of data comparing its effectiveness to emulator tests, we should revisit if/how we want to continue the use of FTL. (In the meantime, this does mean we'll double our exposure to integration test flake for Android, and integration tests are—across platforms—by far the flakiest of tests.)