Add `--ignore-timeouts` flag for `flutter test` command #164437

nilsreichardt · 2025-03-01T22:03:09Z

As in #105913 described, running integration tests of your app often times out. The issue for this is that the test package has their own timeout for loading the test suite:

https://github.com/dart-lang/test/blob/db8cf091506a67eba1e523215e0e49a0db7cd1fd/pkgs/test_core/lib/src/runner/load_suite.dart#L23-L29

This timeout is not configurable. However, you can bypass this timeout using --ignore-timeouts when running dart test. This PR adds the --ignore-timeouts flag to the flutter test command and passes it to the test package.

Adding the flag would be the easiest fix for #105913. I will later add documentation to the integration test docs, that passing this flag will fix the timeout issue. Otherwise, we would need to make the load test suite timeout configurable in the test package and then somehow set it in the flutter test command.

Fixes #105913

A screenshot of running flutter test integration_test --ignore-timeouts which surpasses the previous timeout of 12 minutes.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for submitting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
I signed the CLA.
I listed at least one issue that this PR fixes in the description above.
I updated/added relevant documentation (doc comments with ///).
I added new tests to check the change I am making, or this PR is test-exempt.
I followed the breaking change policy and added Data Driven Fixes where supported.
All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

nilsreichardt · 2025-03-02T11:15:18Z

packages/flutter_tools/lib/src/commands/test.dart

+            'Ignore all timeouts. Useful when testing a big application '
+            'that requires a longer time to compile (e.g. running integration '
+            'tests for a Flutter app).',


I'm not so happy with the helper text, as I assume it's confusing that the timeout help text says 'or as the string "none" to disable the timeout entirely' and a --ignore-timeouts flag, which sounds like the same thing but is not.

I think it might be worth also updating the help text for timeout to explicitly state that it only applies to the timeout for individual tests. Maybe something like this:

The default timeout for individual tests, specified either in seconds (e.g. "60s"), as a multiplier of the default test timeout (e.g. "2x"), or as the string "none" to disable test timeouts entirely. This value does not apply to the default test suite loading timeout.

OliverNarramore · 2025-03-18T10:26:51Z

This would be helpful to us, thank you!

bkonyi

Thanks for the contribution (and your patience)!

This looks good to me overall once my comment is addressed.

I've pinged dart-lang/test#2463 to get some feedback from the package:test maintainers, because there should be a way to configure the test suite loading timeout without having to completely disable timeouts for individual tests.

bkonyi · 2025-03-28T14:16:53Z

packages/flutter_tools/lib/src/commands/test.dart

+            'Ignore all timeouts. Useful when testing a big application '
+            'that requires a longer time to compile (e.g. running integration '
+            'tests for a Flutter app).',


I think it might be worth also updating the help text for timeout to explicitly state that it only applies to the timeout for individual tests. Maybe something like this:

The default timeout for individual tests, specified either in seconds (e.g. "60s"), as a multiplier of the default test timeout (e.g. "2x"), or as the string "none" to disable test timeouts entirely. This value does not apply to the default test suite loading timeout.

bkonyi

Looks good to me! I think you'll need to merge from master and push again to fix the CI error.

nilsreichardt · 2025-03-28T14:57:33Z

Thanks for your review ❤️

Any idea why the CI breaks after the merge?

I executed

git checkout master
git pull
git checkout add-ignore-timeouts-flag # my fix branch
git merge master

Also, after executing git fetch upstream && git merge upstream/master I'm getting:

Already up to date.

bkonyi · 2025-03-28T15:04:20Z

Thanks for your review ❤️

Thank you for your contribution! 😁

Any idea why the CI breaks after the merge?

I executed
git checkout master
git pull
git checkout add-ignore-timeouts-flag # my fix branch
git merge master
Also, after executing git fetch upstream && git merge upstream/master I'm getting:
Already up to date.

That's a good question, I'm not sure... Let me check in with our infra team to see if they have any ideas.

…ts-flag

jyameo

LGTM!

nilsreichardt · 2025-03-28T18:36:22Z

That's a good question, I'm not sure... Let me check in with our infra team to see if they have any ideas.

Since some new commits landed on the master branch, I tried git fetch upstream && git merge upstream/master again, and it works now :)

zanderso · 2025-03-31T22:24:45Z

@bkonyi @jyameo One of you should add the autosubmit label if this is ready to go.

…utter/flutter#164437)

Roll Flutter from 05b5e7910544 to a0b1b3253416 (37 revisions) flutter/flutter@05b5e79...a0b1b32 2025-04-01 [email protected] Roll Dart SDK from b4d374ec59ec to 4e1f02bc704f (2 revisions) (flutter/flutter#166342) 2025-04-01 [email protected] Trim any text before osascript JSON response (flutter/flutter#166296) 2025-04-01 [email protected] [Gen-l10n] Add `Message.resourceId` and `locale` to all `L10nException` error messages (flutter/flutter#163654) 2025-04-01 [email protected] Add `--ignore-timeouts` flag for `flutter test` command (flutter/flutter#164437) 2025-04-01 [email protected] Update TESTOWNERS username (flutter/flutter#166191) 2025-04-01 [email protected] Roll Skia from 4b07443e6071 to 52cbb917fffd (4 revisions) (flutter/flutter#166329) 2025-04-01 [email protected] Roll Dart SDK from 6b07a09cbd2d to b4d374ec59ec (2 revisions) (flutter/flutter#166321) 2025-04-01 [email protected] [tool] Improve using project files in build targets (flutter/flutter#166211) 2025-04-01 [email protected] Rename FlRenderer to FlCompositorOpenGL (flutter/flutter#166037) 2025-04-01 [email protected] [engine, web_ui] Fix instances of library_private_types_in_public_api (flutter/flutter#166156) 2025-04-01 [email protected] Roll Dart SDK from 509faa921c95 to 6b07a09cbd2d (1 revision) (flutter/flutter#166301) 2025-04-01 [email protected] [Impeller] small cpu perf for text contents. (flutter/flutter#166199) 2025-04-01 [email protected] [android_engine_test] disable old HC mode tests. (flutter/flutter#166293) 2025-04-01 [email protected] [impeller] fixes diagonal antialiased lines (flutter/flutter#166298) 2025-04-01 [email protected] Roll Skia from 5f262bd2cbb4 to 4b07443e6071 (10 revisions) (flutter/flutter#166299) 2025-03-31 [email protected] [Impeller] Directly tessellate conics to linear path segments (flutter/flutter#166165) 2025-03-31 [email protected] [tool] Don't write the .flutter-plugins-dependencies file if it is unchanged (flutter/flutter#166164) 2025-03-31 [email protected] Move `.cxx` directory out of `android/app` (flutter/flutter#166277) 2025-03-31 [email protected] Fix typo in carousel.dart (flutter/flutter#164727) 2025-03-31 [email protected] Roll Dart SDK from c5fa06710bb6 to 509faa921c95 (1 revision) (flutter/flutter#166283) 2025-03-31 [email protected] Public nodes needing paint or layout (flutter/flutter#166148) 2025-03-31 [email protected] [Gen-l10n] Infer placeholder types on both templates and localizations (flutter/flutter#163690) 2025-03-31 [email protected] [Engine][iOS] Cancel animation when recieved `UIKeyboardWillHideNotification` with duration 0.0 (flutter/flutter#164884) 2025-03-31 [email protected] [fuchsia] Remove explicit LogSink and InspectSink routing and use dictionaries instead (flutter/flutter#162780) 2025-03-31 [email protected] Updated to latest AVD to Support Android 16 (API 36) (flutter/flutter#165926) 2025-03-31 [email protected] Feat: Add brightnessOf method for theme (flutter/flutter#163733) 2025-03-31 [email protected] Marks Linux_mokey new_gallery__crane_perf to be flaky (flutter/flutter#165964) 2025-03-31 [email protected] [ Tool ] Correctly select entrypoint target for web build from positional argument list (flutter/flutter#166260) 2025-03-31 [email protected] [Impeller] remove validation warning ignores. (flutter/flutter#166205) 2025-03-31 [email protected] [Impeller] handle shader ordering bug on macOS. (flutter/flutter#165937) 2025-03-31 [email protected] Fix CODEOWNERS for the iOS review team (flutter/flutter#166178) 2025-03-31 [email protected] Remove `<meta content="IE=Edge" http-equiv="X-UA-Compatible">` (flutter/flutter#166252) 2025-03-31 [email protected] Roll Dart SDK from b9c35e05feb5 to c5fa06710bb6 (1 revision) (flutter/flutter#166251) 2025-03-31 [email protected] Roll Skia from 418c68ea5ccb to 5f262bd2cbb4 (2 revisions) (flutter/flutter#166244) 2025-03-31 [email protected] Roll Skia from b6a3bbd1d153 to 418c68ea5ccb (1 revision) (flutter/flutter#166236) 2025-03-31 [email protected] [Impeller] fix min filter for GL external textures. (flutter/flutter#166224) 2025-03-31 [email protected] Roll Skia from 10f4cf9a817d to b6a3bbd1d153 (13 revisions) (flutter/flutter#166231) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose ...

…r#8985) Roll Flutter from 05b5e7910544 to a0b1b3253416 (37 revisions) flutter/flutter@05b5e79...a0b1b32 2025-04-01 [email protected] Roll Dart SDK from b4d374ec59ec to 4e1f02bc704f (2 revisions) (flutter/flutter#166342) 2025-04-01 [email protected] Trim any text before osascript JSON response (flutter/flutter#166296) 2025-04-01 [email protected] [Gen-l10n] Add `Message.resourceId` and `locale` to all `L10nException` error messages (flutter/flutter#163654) 2025-04-01 [email protected] Add `--ignore-timeouts` flag for `flutter test` command (flutter/flutter#164437) 2025-04-01 [email protected] Update TESTOWNERS username (flutter/flutter#166191) 2025-04-01 [email protected] Roll Skia from 4b07443e6071 to 52cbb917fffd (4 revisions) (flutter/flutter#166329) 2025-04-01 [email protected] Roll Dart SDK from 6b07a09cbd2d to b4d374ec59ec (2 revisions) (flutter/flutter#166321) 2025-04-01 [email protected] [tool] Improve using project files in build targets (flutter/flutter#166211) 2025-04-01 [email protected] Rename FlRenderer to FlCompositorOpenGL (flutter/flutter#166037) 2025-04-01 [email protected] [engine, web_ui] Fix instances of library_private_types_in_public_api (flutter/flutter#166156) 2025-04-01 [email protected] Roll Dart SDK from 509faa921c95 to 6b07a09cbd2d (1 revision) (flutter/flutter#166301) 2025-04-01 [email protected] [Impeller] small cpu perf for text contents. (flutter/flutter#166199) 2025-04-01 [email protected] [android_engine_test] disable old HC mode tests. (flutter/flutter#166293) 2025-04-01 [email protected] [impeller] fixes diagonal antialiased lines (flutter/flutter#166298) 2025-04-01 [email protected] Roll Skia from 5f262bd2cbb4 to 4b07443e6071 (10 revisions) (flutter/flutter#166299) 2025-03-31 [email protected] [Impeller] Directly tessellate conics to linear path segments (flutter/flutter#166165) 2025-03-31 [email protected] [tool] Don't write the .flutter-plugins-dependencies file if it is unchanged (flutter/flutter#166164) 2025-03-31 [email protected] Move `.cxx` directory out of `android/app` (flutter/flutter#166277) 2025-03-31 [email protected] Fix typo in carousel.dart (flutter/flutter#164727) 2025-03-31 [email protected] Roll Dart SDK from c5fa06710bb6 to 509faa921c95 (1 revision) (flutter/flutter#166283) 2025-03-31 [email protected] Public nodes needing paint or layout (flutter/flutter#166148) 2025-03-31 [email protected] [Gen-l10n] Infer placeholder types on both templates and localizations (flutter/flutter#163690) 2025-03-31 [email protected] [Engine][iOS] Cancel animation when recieved `UIKeyboardWillHideNotification` with duration 0.0 (flutter/flutter#164884) 2025-03-31 [email protected] [fuchsia] Remove explicit LogSink and InspectSink routing and use dictionaries instead (flutter/flutter#162780) 2025-03-31 [email protected] Updated to latest AVD to Support Android 16 (API 36) (flutter/flutter#165926) 2025-03-31 [email protected] Feat: Add brightnessOf method for theme (flutter/flutter#163733) 2025-03-31 [email protected] Marks Linux_mokey new_gallery__crane_perf to be flaky (flutter/flutter#165964) 2025-03-31 [email protected] [ Tool ] Correctly select entrypoint target for web build from positional argument list (flutter/flutter#166260) 2025-03-31 [email protected] [Impeller] remove validation warning ignores. (flutter/flutter#166205) 2025-03-31 [email protected] [Impeller] handle shader ordering bug on macOS. (flutter/flutter#165937) 2025-03-31 [email protected] Fix CODEOWNERS for the iOS review team (flutter/flutter#166178) 2025-03-31 [email protected] Remove `<meta content="IE=Edge" http-equiv="X-UA-Compatible">` (flutter/flutter#166252) 2025-03-31 [email protected] Roll Dart SDK from b9c35e05feb5 to c5fa06710bb6 (1 revision) (flutter/flutter#166251) 2025-03-31 [email protected] Roll Skia from 418c68ea5ccb to 5f262bd2cbb4 (2 revisions) (flutter/flutter#166244) 2025-03-31 [email protected] Roll Skia from b6a3bbd1d153 to 418c68ea5ccb (1 revision) (flutter/flutter#166236) 2025-03-31 [email protected] [Impeller] fix min filter for GL external textures. (flutter/flutter#166224) 2025-03-31 [email protected] Roll Skia from 10f4cf9a817d to b6a3bbd1d153 (13 revisions) (flutter/flutter#166231) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose ...

Add --ignore-timeouts flag for flutter test command

17d9f3a

github-actions bot added the tool Affects the "flutter" command-line tool. See also t: labels. label Mar 1, 2025

Fix passing --ignore-timeouts by default

598af15

nilsreichardt commented Mar 2, 2025

View reviewed changes

nilsreichardt mentioned this pull request Mar 2, 2025

Integration test TimeoutException after 0:12:00.000000: Test timed out after 12 minutes #105913

Closed

bkonyi reviewed Mar 28, 2025

View reviewed changes

Clarify the --timeout flag

a3cb3f3

bkonyi requested a review from jyameo March 28, 2025 14:33

bkonyi approved these changes Mar 28, 2025

View reviewed changes

Merge branch 'master' into add-ignore-timeouts-flag

9f9be46

Merge remote-tracking branch 'upstream/master' into add-ignore-timeou…

3fb9bc8

…ts-flag

jyameo approved these changes Mar 28, 2025

View reviewed changes

jyameo added the autosubmit Merge PR when tree becomes green via auto submit App label Apr 1, 2025

auto-submit bot added this pull request to the merge queue Apr 1, 2025

Merged via the queue into flutter:master with commit 70cd54c Apr 1, 2025
136 checks passed

flutter-dashboard bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Apr 1, 2025

engine-flutter-autoroll mentioned this pull request Apr 1, 2025

Roll Flutter from 1d954f4e96bd to 70cd54c3fd13 (259 revisions) flutter/packages#8973

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

292ed2d

…utter/flutter#164437)

engine-flutter-autoroll mentioned this pull request Apr 1, 2025

Roll Flutter from 1d954f4e96bd to 0befaaafc952 (260 revisions) flutter/packages#8974

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

1a75177

…utter/flutter#164437)

engine-flutter-autoroll mentioned this pull request Apr 1, 2025

Roll Flutter from 1d954f4e96bd to dbef1314b186 (261 revisions) flutter/packages#8976

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

8b49367

…utter/flutter#164437)

engine-flutter-autoroll mentioned this pull request Apr 1, 2025

Roll Flutter from 1d954f4e96bd to a0b1b3253416 (262 revisions) flutter/packages#8977

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

bdbcc22

…utter/flutter#164437)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 2, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

9d398d5

…utter/flutter#164437)

This was referenced Apr 2, 2025

Roll Flutter from 05b5e7910544 to a0b1b3253416 (37 revisions) flutter/packages#8983

Closed

Roll Flutter from 05b5e7910544 to a0b1b3253416 (37 revisions) flutter/packages#8985

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 2, 2025

70cd54c3f Add --ignore-timeouts flag for flutter test command (fl…

21dbf4e

…utter/flutter#164437)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `--ignore-timeouts` flag for `flutter test` command #164437

Add `--ignore-timeouts` flag for `flutter test` command #164437

nilsreichardt commented Mar 1, 2025 •

edited

Loading

nilsreichardt Mar 2, 2025

bkonyi Mar 28, 2025

OliverNarramore commented Mar 18, 2025

bkonyi left a comment

bkonyi Mar 28, 2025

bkonyi left a comment

nilsreichardt commented Mar 28, 2025

bkonyi commented Mar 28, 2025

jyameo left a comment

nilsreichardt commented Mar 28, 2025

zanderso commented Mar 31, 2025

Add --ignore-timeouts flag for flutter test command #164437

Add --ignore-timeouts flag for flutter test command #164437

Conversation

nilsreichardt commented Mar 1, 2025 • edited Loading

Pre-launch Checklist

nilsreichardt Mar 2, 2025

Choose a reason for hiding this comment

bkonyi Mar 28, 2025

Choose a reason for hiding this comment

OliverNarramore commented Mar 18, 2025

bkonyi left a comment

Choose a reason for hiding this comment

bkonyi Mar 28, 2025

Choose a reason for hiding this comment

bkonyi left a comment

Choose a reason for hiding this comment

nilsreichardt commented Mar 28, 2025

bkonyi commented Mar 28, 2025

jyameo left a comment

Choose a reason for hiding this comment

nilsreichardt commented Mar 28, 2025

zanderso commented Mar 31, 2025

Add `--ignore-timeouts` flag for `flutter test` command #164437

Add `--ignore-timeouts` flag for `flutter test` command #164437

nilsreichardt commented Mar 1, 2025 •

edited

Loading