Randomize Framework tests, opt out some tests that currently fail. #85159
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
flutter-dashboard
bot
added
f: cupertino
gspencergoog
force-pushed
the
randomize_tests
branch
2 times, most recently
from
d493f92 to
fd19a4b
Compare
gspencergoog
force-pushed
the
randomize_tests
branch
8 times, most recently
from
f443cc6 to
9fcfda3
Compare
gspencergoog
force-pushed
the
randomize_tests
branch
3 times, most recently
from
22b6075 to
0dab346
Compare
Contributor
darrenaustin
left a comment
darrenaustin
left a comment
There was a problem hiding this comment.
Nice work. A few questions below.
gspencergoog
force-pushed
the
randomize_tests
branch
2 times, most recently
from
99e229d to
2556a4f
Compare
gspencergoog
force-pushed
the
randomize_tests
branch
from
2556a4f to
0dc0332
Compare
gspencergoog
added a commit
to gspencergoog/flutter
that referenced
this pull request
Jul 1, 2021
…fail. (flutter#85159)" This reverts commit b5f9612 because it is taking about 40% longer to run the tests, which is causing timeouts.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This turns on order shuffling for all tests that don't fail with it on, marking those tests that do fail with a tag so that they will be run without shuffling on.
To determine which tests fail with it on, I ran all the tests 100 times with different random shuffle seeds, and then also ran it with the date seeds from today until the end of July, and tagged all of the test suites (files) that fail, with a seed that caused them to fail.
Related Issues
no-shuffletags from all framework tests #85160Tests