Skip to content

[integration_test] Changing test code doesn't work with --no-build parameter #72276

Closed as duplicate

Description

@vrnvorona

Use case

Writing and debugging tests become time consuming with ~60 seconds build time. If i pass --no-build parameter with my flutter drive command, code changes doesn't work as if tests are baked into build.

Steps to reproduce are pretty simple:

  1. Get example integration_test
  2. Run it in typical flutter drive fashion
  3. Edit test code, say expect(2 + 2, equals(4)) instead of 5 for a good measure
  4. Run test with flutter drive --target --driver --no-build command (with respective dart files for target and driver)

Expected result: Test passes (assuming test at second step was expecting 5 and failed always)
Actual result: Test fails, because it still expects 5

Example logs where I ran 2+2=5 test 3 times, in second run I changed it to expect 4, and in third I rebuilt app ➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart Using device iPhone 8 Plus. Starting application: test_driver/test_fail.dart Warning: Missing build name (CFBundleShortVersionString). Warning: Missing build number (CFBundleVersion). Action Required: You must set a build name and number in the pubspec.yaml file version field before submitting to the App Store. Running Xcode build... └─Compiling, linking and signing... 4.1s Xcode build done. 15.9s flutter: Observatory listening on http://127.0.0.1:56138/dxgRaQUBub0=/ VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56138/dxgRaQUBub0=/ VMServiceFlutterDriver: Isolate found with number: 1817371241983551 VMServiceFlutterDriver: Isolate is paused at start. VMServiceFlutterDriver: Attempting to resume isolate flutter: 00:00 +0: end-to-end test failing test example VMServiceFlutterDriver: Connected to Flutter application. flutter: (The following exception is now available via WidgetTester.takeException:) flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ flutter: The following TestFailure object was thrown running a test: flutter: Expected: <5> flutter: Actual: <4> flutter: flutter: When the exception was thrown, this was the stack: flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7) flutter: flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart) flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29) flutter: flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart) flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19) flutter: flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14) flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12) flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17) flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24) flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19) flutter: flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart) flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13) flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15) flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5) flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17) flutter: flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart) flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9) flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15) flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7) flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11) flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11) flutter: (elided 31 frames from dart:async and package:stack_trace) flutter: flutter: This was caught by the test expectation on the following line: flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15 flutter: The test description was: flutter: failing test example flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════ flutter: (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens first, and then the test will fail due to having not caught or expected the exception.) flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ flutter: The following TestFailure object was thrown running a test: flutter: Expected: <5> flutter: Actual: <4> flutter: flutter: When the exception was thrown, this was the stack: flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7) flutter: flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart) flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29) flutter: flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart) flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19) flutter: flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14) flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12) flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17) flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24) flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19) flutter: flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart) flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13) flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15) flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5) flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17) flutter: flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart) flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9) flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15) flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7) flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11) flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11) flutter: (elided 31 frames from dart:async and package:stack_trace) flutter: flutter: This was caught by the test expectation on the following line: flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15 flutter: The test description was: flutter: failing test example flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════ flutter: 00:00 +0: end-to-end test failing test example [E] flutter: Test failed. See exception logs above. The test description was: failing test example flutter: flutter: 00:00 +0 -1: end-to-end test (tearDownAll) Failure Details: Failure in method: failing test example ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════ The following TestFailure object was thrown running a test: Expected: <5> Actual: <4>

When the exception was thrown, this was the stack:
#4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)

#5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
#6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)

#7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
#8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)

#11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
#12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
#13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
#14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
#15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)

#16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
#21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
#22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
#27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
#28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)

#29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
#34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
#35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
#36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
#43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
#44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
(elided 31 frames from dart:async and package:stack_trace)

This was caught by the test expectation on the following line:
file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
The test description was:
failing test example
═════════════════════════════════════════════════════════════════

end of failure 1

Stopping application instance.
Driver tests failed: 1
➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart --no-build
Using device iPhone 8 Plus.
Starting application: test_driver/test_fail.dart
flutter: Observatory listening on http://127.0.0.1:56193/8dbtOUPbkk4=/
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56193/8dbtOUPbkk4=/
VMServiceFlutterDriver: Isolate found with number: 4072074419452431
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: end-to-end test failing test example
VMServiceFlutterDriver: Connected to Flutter application.
flutter: (The following exception is now available via WidgetTester.takeException:)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter: Expected: <5>
flutter: Actual: <4>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
flutter:
flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
flutter:
flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
flutter:
flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
flutter:
flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
flutter:
flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
flutter: The test description was:
flutter: failing test example
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: (If WidgetTester.takeException is called, the above exception will be ignored. If it is not, then the above exception will be dumped when another exception is caught by the framework or when the test ends, whichever happens
first, and then the test will fail due to having not caught or expected the exception.)
flutter: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
flutter: The following TestFailure object was thrown running a test:
flutter: Expected: <5>
flutter: Actual: <4>
flutter:
flutter: When the exception was thrown, this was the stack:
flutter: #4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)
flutter:
flutter: #5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
flutter: #6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)
flutter:
flutter: #7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
flutter: #8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)
flutter:
flutter: #11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
flutter: #12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
flutter: #13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
flutter: #14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
flutter: #15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)
flutter:
flutter: #16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
flutter: #21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
flutter: #22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
flutter: #27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
flutter: #28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)
flutter:
flutter: #29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
flutter: #34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
flutter: #35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
flutter: #36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
flutter: #43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
flutter: #44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
flutter: (elided 31 frames from dart:async and package:stack_trace)
flutter:
flutter: This was caught by the test expectation on the following line:
flutter: file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
flutter: The test description was:
flutter: failing test example
flutter: ════════════════════════════════════════════════════════════════════════════════════════════════════
flutter: 00:00 +0: end-to-end test failing test example [E]
flutter: Test failed. See exception logs above.
The test description was: failing test example
flutter:
flutter: 00:00 +0 -1: end-to-end test (tearDownAll)
flutter: Warning: integration_test test plugin was not detected.
flutter: 00:00 +1 -1: Some tests failed.
Failure Details:
Failure in method: failing test example
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following TestFailure object was thrown running a test:
Expected: <5>
Actual: <4>

When the exception was thrown, this was the stack:
#4 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart:15:7)

#5 main.. (file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart)
#6 testWidgets.. (package:flutter_test/src/widget_tester.dart:146:29)

#7 testWidgets.. (package:flutter_test/src/widget_tester.dart)
#8 TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:786:19)

#11 TestWidgetsFlutterBinding._runTest (package:flutter_test/src/binding.dart:766:14)
#12 LiveTestWidgetsFlutterBinding.runTest (package:flutter_test/src/binding.dart:1596:12)
#13 IntegrationTestWidgetsFlutterBinding.runTest (package:integration_test/integration_test.dart:194:17)
#14 testWidgets. (package:flutter_test/src/widget_tester.dart:138:24)
#15 Declarer.test.. (package:test_api/src/backend/declarer.dart:175:19)

#16 Declarer.test.. (package:test_api/src/backend/declarer.dart)
#21 Declarer.test. (package:test_api/src/backend/declarer.dart:173:13)
#22 Invoker.waitForOutstandingCallbacks. (package:test_api/src/backend/invoker.dart:231:15)
#27 Invoker.waitForOutstandingCallbacks (package:test_api/src/backend/invoker.dart:228:5)
#28 Invoker._onRun... (package:test_api/src/backend/invoker.dart:383:17)

#29 Invoker._onRun... (package:test_api/src/backend/invoker.dart)
#34 Invoker._onRun.. (package:test_api/src/backend/invoker.dart:370:9)
#35 Invoker._guardIfGuarded (package:test_api/src/backend/invoker.dart:415:15)
#36 Invoker._onRun. (package:test_api/src/backend/invoker.dart:369:7)
#43 Invoker._onRun (package:test_api/src/backend/invoker.dart:368:11)
#44 LiveTestController.run (package:test_api/src/backend/live_test_controller.dart:153:11)
(elided 31 frames from dart:async and package:stack_trace)

This was caught by the test expectation on the following line:
file:///Users/voronin/plugins/packages/integration_test/example/test_driver/test_fail.dart line 15
The test description was:
failing test example
═════════════════════════════════════════════════════════════════

end of failure 1

Stopping application instance.
Driver tests failed: 1
➜ example git:(integration_test_test) ✗ flutter drive --driver=test_driver/integration_test.dart --target=test_driver/test_fail.dart
Using device iPhone 8 Plus.
Starting application: test_driver/test_fail.dart
Warning: Missing build name (CFBundleShortVersionString).
Warning: Missing build number (CFBundleVersion).
Action Required: You must set a build name and number in the pubspec.yaml file version field before submitting to the App Store.
Running Xcode build...
└─Compiling, linking and signing... 4.2s
Xcode build done. 15.9s
flutter: Observatory listening on http://127.0.0.1:56243/N-FWUbl_mNc=/
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:56243/N-FWUbl_mNc=/
VMServiceFlutterDriver: Isolate found with number: 2612504707691895
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: end-to-end test failing test example
VMServiceFlutterDriver: Connected to Flutter application.
flutter: 00:00 +1: end-to-end test (tearDownAll)
flutter: Warning: integration_test test plugin was not detected.
flutter: 00:00 +2: All tests passed!
All tests passed.
Stopping application instance.

Proposal

Somehow, if possible, allow for changing code without re-build of app under the test. Previous tool for integration testing (infamous flutter_driver) allowed such things.

Activity

  1. nmtitov commented on Dec 30, 2020

    @nmtitov

    This would be really useful for TDD development. I also noticed that running integration_tests required rebuilding the iOS project but didn't expect it to be considered a bug until I've found this thread.

  2. slavap commented on Apr 30, 2021

    @slavap

    Actually --no-build does not work for integration tests, and situation is especially BAD for Flutter Web, because for Android there is working --use-application-binary flag.

    See #68529

  3. added
    frameworkflutter/packages/flutter repository. See also f: labels.
    on Apr 21, 2022
  4. removed
    frameworkflutter/packages/flutter repository. See also f: labels.
    on Jun 16, 2022
  5. bartekpacia commented on Aug 27, 2022

    @bartekpacia
    Member

    It's be great to see someone from the Flutter team say something about this.

  6. added
    toolAffects the "flutter" command-line tool. See also t: labels.
    on Jan 27, 2023
  7. added
    P3Issues that are less important to the Flutter project
    triaged-toolTriaged by Flutter Tool team
    on Oct 16, 2023
  8. github-actions commented on Jul 24, 2025

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  9. locked as resolved and limited conversation to collaborators on Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P3Issues that are less important to the Flutter projecta: tests"flutter test", flutter_test, or one of our testsc: proposalA detailed proposal for a change to Flutterf: integration_testThe flutter/packages/integration_test pluginteam-toolOwned by Flutter Tool teamtoolAffects the "flutter" command-line tool. See also t: labels.triaged-toolTriaged by Flutter Tool team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions