Synopsis

Currently flutter test integration_test builds a new .apk (Android) / .app (iOS) for every integration test file. The problem with this approach is that it greatly slows down the development of integration tests and impedes developer/tester experience. As a Flutter developer, I'm used to the awesome hot-{reload,restart} feature and the incredibly fast iteration it allows for, and am disappointed that the experience of developing integration tests is very much subpar compared to the development of user code.

Another problem with the way flutter test integration_test works currently is the inability to build integration tests once and then run them multiple times (because every integration test comes in a new app). This feature would be useful in detecting flaky tests faster, without the need to build the application binary every time before running the test file that hasn't been changed in the last N runs.

Description

For example, let's say that our app has 3 integration tests:

integration_test
├── sign_up_test.dart
├── location_test.dart
└── sign_out_test.dart

Running flutter test integration_test builds the application 3 times, once for every integration test file:

  > sign_up_test.dart
    > Build app
    > Install app
    > Run app and execute tests
    > Kill app
    > Uninstall app
  > location_test.dart
    > Build app
    > Install app
    > Run app and execute tests
    > Kill app
    > Uninstall app
  > sign_out_test.dart
    > Build app
    > Install app
    > Run app and execute tests
    > Kill app
    > Uninstall app

This seems unnecessary because almost always, the only thing that differs between these applications is the integration test file, which is often 100-200 lines of code.

Let me stress this: to apply a change of a 100-200 lines of test code, users have to rebuild the whole application.

I think it'd be reasonable if running flutter test integration_test on the aforementioned application with 3 integration tests worked like this:

  > Build app (with all tests compiled into it)
  > Install app
  > Run app with sign_up_test.dart as main()
  > Clear app data
  > Run app with location_test.dart as main()
  > Clear app data
  > Run app with sign_out_test.dart as main()
  > Clear app data
  > No more tests to run, uninstall app

The idea for this issue comes from this comment by @jonahwilliams.

Issues that seem to be related:

• flutter test is missing the --use-application-binary argument, which flutter drive has #114541
• Rerunning the same flutter test test on a target device causes a full rebuild every time #108400
• [integration_test] Enable sharding of integration test suites #101296
• Hot reload with integration_test  #76985

$ flutter --version
Flutter 3.3.7 • channel stable • https://github.com/flutter/flutter.git
Framework • revision e99c9c7cd9 (3 weeks ago) • 2022-11-01 16:59:00 -0700
Engine • revision 857bd6b74c
Tools • Dart 2.18.4 • DevTools 2.15.0