Repository navigation
Description
Use case
Writing and debugging tests become time consuming with ~60 seconds build time. If i pass --no-build parameter with my flutter drive command, code changes doesn't work as if tests are baked into build.
Steps to reproduce are pretty simple:
- Get example integration_test
- Run it in typical
flutter drivefashion - Edit test code, say
expect(2 + 2, equals(4))instead of 5 for a good measure - Run test with
flutter drive --target --driver --no-buildcommand (with respective dart files for target and driver)
Expected result: Test passes (assuming test at second step was expecting 5 and failed always)
Actual result: Test fails, because it still expects 5
Example logs where I ran 2+2=5 test 3 times, in second run I changed it to expect 4, and in third I rebuilt app
➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart Using device iPhone 8 Plus. Starting application: test_driver/test_fail.dart Warning: Missing build name (CFBundleShortVersionString). Warning: Missing build number (CFBundleVersion). Action Required: You must set a build name and number in the pubspec.yaml file version field before submitting to the App Store. Running Xcode build... └─Compiling, linking and signing... 4.1s Xcode build done. 15.9s flutter: Observatory listening on http://127.0.0.1:56138/dxgRaQUBub0=/ VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56138/dxgRaQUBub0=/ VMServiceFlutterDriver: Isolate found with number: 1817371241983551 VMServiceFlutterDriver: Isolate is paused at start. VMServiceFlutterDriver: Attempting to resume isolate flutter: 00:00 +0: end-to-end test failing test example VMServiceFlutterDriver: Connected to Flutter application. flutter: (The following exception is now available via WidgetTester.takeException:) flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ flutter: The following TestFailure object was thrown running a test: flutter: Expected: <5> flutter: Actual: <4> flutter: flutter: When the exception was thrown, this was the stack: flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7) flutter: flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart) flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29) flutter: flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart) flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19) flutter: flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14) flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12) flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17) flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24) flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19) flutter: flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart) flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13) flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15) flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5) flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17) flutter: flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart) flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9) flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15) flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7) flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11) flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11) flutter: (elided 31 frames from dart:async and package:stack_trace) flutter: flutter: This was caught by the test expectation on the following line: flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15 flutter: The test description was: flutter: failing test example flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════ flutter: (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens first, and then the test will fail due to having not caught or expected the exception.) flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ flutter: The following TestFailure object was thrown running a test: flutter: Expected: <5> flutter: Actual: <4> flutter: flutter: When the exception was thrown, this was the stack: flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7) flutter: flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart) flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29) flutter: flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart) flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19) flutter: flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14) flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12) flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17) flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24) flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19) flutter: flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart) flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13) flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15) flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5) flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17) flutter: flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart) flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9) flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15) flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7) flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11) flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11) flutter: (elided 31 frames from dart:async and package:stack_trace) flutter: flutter: This was caught by the test expectation on the following line: flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15 flutter: The test description was: flutter: failing test example flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════ flutter: 00:00 +0: end-to-end test failing test example [E] flutter: Test failed. See exception logs above. The test description was: failing test example flutter: flutter: 00:00 +0 -1: end-to-end test (tearDownAll) Failure Details: Failure in method: failing test example ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════ The following TestFailure object was thrown running a test: Expected: <5> Actual: <4>When the exception was thrown, this was the stack:
#4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
#5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
#6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
#7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
#8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
#11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
#12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
#13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
#14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
#15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
#16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
#21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
#22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
#27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
#28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
#29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
#34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
#35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
#36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
#43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
#44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
(elided 31 frames from dart:async and package:stack_trace)
This was caught by the test expectation on the following line:
file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
The test description was:
failing test example
═════════════════════════════════════════════════════════════════
end of failure 1
Stopping application instance.
Driver tests failed: 1
➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart --no-build
Using device iPhone 8 Plus.
Starting application: test_driver/test_fail.dart
flutter: Observatory listening on http://127.0.0.1:56193/8dbtOUPbkk4=/
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56193/8dbtOUPbkk4=/
VMServiceFlutterDriver: Isolate found with number: 4072074419452431
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: end-to-end test failing test example
VMServiceFlutterDriver: Connected to Flutter application.
flutter: (The following exception is now available via WidgetTester.takeException:)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter: Expected: <5>
flutter: Actual: <4>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
flutter:
flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
flutter:
flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
flutter:
flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
flutter:
flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
flutter:
flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
flutter: The test description was:
flutter: failing test example
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens
first, and then the test will fail due to having not caught or expected the exception.)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter: Expected: <5>
flutter: Actual: <4>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
flutter:
flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
flutter:
flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
flutter:
flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
flutter:
flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
flutter:
flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
flutter: The test description was:
flutter: failing test example
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 00:00 +0: end-to-end test failing test example [E]
flutter: Test failed. See exception logs above.
The test description was: failing test example
flutter:
flutter: 00:00 +0 -1: end-to-end test (tearDownAll)
flutter: Warning: integration_test test plugin was not detected.
flutter: 00:00 +1 -1: Some tests failed.
Failure Details:
Failure in method: failing test example
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following TestFailure object was thrown running a test:
Expected: <5>
Actual: <4>
When the exception was thrown, this was the stack:
#4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
#5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
#6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
#7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
#8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
#11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
#12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
#13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
#14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
#15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
#16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
#21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
#22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
#27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
#28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
#29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
#34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
#35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
#36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
#43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
#44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
(elided 31 frames from dart:async and package:stack_trace)
This was caught by the test expectation on the following line:
file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
The test description was:
failing test example
═════════════════════════════════════════════════════════════════
end of failure 1
Stopping application instance.
Driver tests failed: 1
➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart
Using device iPhone 8 Plus.
Starting application: test_driver/test_fail.dart
Warning: Missing build name (CFBundleShortVersionString).
Warning: Missing build number (CFBundleVersion).
Action Required: You must set a build name and number in the pubspec.yaml file version field before submitting to the App Store.
Running Xcode build...
└─Compiling, linking and signing... 4.2s
Xcode build done. 15.9s
flutter: Observatory listening on http://127.0.0.1:56243/N-FWUbl_mNc=/
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56243/N-FWUbl_mNc=/
VMServiceFlutterDriver: Isolate found with number: 2612504707691895
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: end-to-end test failing test example
VMServiceFlutterDriver: Connected to Flutter application.
flutter: 00:00 +1: end-to-end test (tearDownAll)
flutter: Warning: integration_test test plugin was not detected.
flutter: 00:00 +2: All tests passed!
All tests passed.
Stopping application instance.
Proposal
Somehow, if possible, allow for changing code without re-build of app under the test. Previous tool for integration testing (infamous flutter_driver) allowed such things.
Activity
nmtitov commentedon Dec 30, 2020
This would be really useful for TDD development. I also noticed that running integration_tests required rebuilding the iOS project but didn't expect it to be considered a bug until I've found this thread.
slavap commentedon Apr 30, 2021
Actually --no-build does not work for integration tests, and situation is especially BAD for Flutter Web, because for Android there is working --use-application-binary flag.
See #68529
bartekpacia commentedon Aug 27, 2022
It's be great to see someone from the Flutter team say something about this.
github-actions commentedon Jul 24, 2025
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.