Sometimes integration test fails to start on iOS simulator and flutter test takes 12 minutes to time out #153433
Description
Steps to reproduce
flutter test integration_testusing iOS simulator
Expected results
Test runs.
If test fails to load to the phone, then the command exits with an error straight away.
Actual results
About 10% of the time the simulator fails to load the app. Then the test command takes 12 minutes to time out.
I was able to reproduce it both in the CircleCI cloud runners and on a local M1 Mac Mini.
I think it would be good enough if instead of timing out, the test command exited immediately on error. Then I can set up some retry logic in a script that runs the tests.
Code sample
Code sample
[Paste your code here]Screenshots or Video
Screenshots / Video demonstration
[Upload media here]
Logs
Logs
flutter test --flavor production integration_test
00:04 +0: loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart Ru00:25 +0: loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart
00:28 +0: loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart 3.1s
Xcode build done. 23.7s
00:29 +0: loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart Error: Unable to launch my.app.id on F7AF702A-AA98-48AF-989A-60AAA5A3929C:
ProcessException: Process exited abnormally with exit code 4:
An error was encountered processing the command (domain=FBSOpenApplicationServiceErrorDomain, code=4):
Simulator device failed to launch my.app.id.
Underlying error (domain=FBSOpenApplicationServiceErrorDomain, code=4):
The request to open "my.app.id" failed.
Command: /usr/bin/arch -arm64e xcrun simctl launch F7AF702A-AA98-48AF-989A-60AAA5A3929C my.app.id --enable-dart-profiling --disable-vm-service-publication --enable-checked-mode
--verify-entry-points
00:29 +0 -1: loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart [E]
TestDeviceException(Unable to start the app on the device.)
package:flutter_tools/src/test/integration_test_device.dart 63:7 IntegrationTestTestDevice.start
TimeoutException after 0:12:00.000000: Test timed out after 12 minutes.
package:test_api/src/backend/invoker.dart 338:28 Invoker._handleError.<fn>
dart:async/zone.dart 1391:47 _rootRun
dart:async/zone.dart 1301:19 _CustomZone.run
package:test_api/src/backend/invoker.dart 336:10 Invoker._handleError
package:test_api/src/backend/invoker.dart 291:9 Invoker.heartbeat.<fn>.<fn>
dart:async/zone.dart 1399:13 _rootRun
dart:async/zone.dart 1301:19 _CustomZone.run
package:test_api/src/backend/invoker.dart 290:38 Invoker.heartbeat.<fn>
dart:async-patch/timer_patch.dart 18:15 Timer._createTimer.<fn>
dart:isolate-patch/timer_impl.dart 398:19 _Timer._runTimers
dart:isolate-patch/timer_impl.dart 429:5 _Timer._handleMessage
dart:isolate-patch/isolate_patch.dart 184:12 _RawReceivePort._handleMessage
To run this test again: /Users/me/development/flutter/bin/cache/dart-sdk/bin/dart test /Users/me/projects/myapp/integration_test/database/database_interface_test.dart -p vm --plain-name 'loading /Users/me/projects/myapp/integration_test/database/database_interface_test.dart'
12:04 +0 -1: loading /Users/me/projects/myapp/integration_test/database/native_migrations_test.dart Ru12:24 +0 -1: loading /Users/me/projects/myapp/integration_test/database/native_migrations_test.dart
12:26 +0 -1: loading /Users/me/projects/myapp/integration_test/database/native_migrations_test.dart 1,959ms
Xcode build done. 21.9s
12:33 +0 ~11 -1: Some tests failed. Flutter Doctor output
Doctor output
[✓] Flutter (Channel stable, 3.22.3, on macOS 14.4.1 23E224 darwin-arm64, locale en-GB)
• Flutter version 3.22.3 on channel stable at /Users/me/development/flutter
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision b0850beeb2 (4 weeks ago), 2024-07-16 21:43:41 -0700
• Engine revision 235db911ba
• Dart version 3.4.4
• DevTools version 2.34.3
[✗] Android toolchain - develop for Android devices
✗ Unable to locate Android SDK.
Install Android Studio from: https://developer.android.com/studio/index.html
On first launch it will assist you in installing the Android SDK components.
(or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
If the Android SDK has been installed to a custom location, please use
`flutter config --android-sdk` to update to that location.
[✓] Xcode - develop for iOS and macOS (Xcode 15.4)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Build 15F31d
• CocoaPods version 1.15.2
[✗] Chrome - develop for the web (Cannot find Chrome executable at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome)
! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[!] Android Studio (not installed)
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/docs/get-started/install/macos#android-setup for detailed instructions).
[✓] Connected device (3 available)
• test (mobile) • F7AF702A-AA98-48AF-989A-60AAA5A3929C • ios • com.apple.CoreSimulator.SimRuntime.iOS-16-4 (simulator)
• macOS (desktop) • macos • darwin-arm64 • macOS 14.4.1 23E224 darwin-arm64
• Mac Designed for iPad (desktop) • mac-designed-for-ipad • darwin • macOS 14.4.1 23E224 darwin-arm64
[✓] Network resources
• All expected network resources are available.
! Doctor found issues in 3 categories.