Repository navigation
Closed
Labels
P2Important issues not at the top of the work listImportant issues not at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.Team-specific productivity, code health, technical debt.c: tech-debtTechnical debt, code quality, testing, etc.Technical debt, code quality, testing, etc.team-toolOwned by Flutter Tool teamOwned by Flutter Tool teamtoolAffects the "flutter" command-line tool. See also t: labels.Affects the "flutter" command-line tool. See also t: labels.tool-still-validIssues that have been deemed still valid as part of the Flutter Tools issue cleanup initiative.Issues that have been deemed still valid as part of the Flutter Tools issue cleanup initiative.triaged-toolTriaged by Flutter Tool teamTriaged by Flutter Tool team
Description
Description
In #86793 we turned on shuffling of test order based on a seed in order to root out test state leakage. In the process, we added a testing tag called 'no-shuffle' which was applied to the majority of tests which currently fail with the shuffling turned on.
This issue is to clean up all of those instances of broken tests, and finally to remove the "no-shuffle" tag entirely to turn on shuffling for all tests.
Related issues
Metadata
Metadata
Assignees
Labels
P2Important issues not at the top of the work listImportant issues not at the top of the work listc: contributor-productivityTeam-specific productivity, code health, technical debt.Team-specific productivity, code health, technical debt.c: tech-debtTechnical debt, code quality, testing, etc.Technical debt, code quality, testing, etc.team-toolOwned by Flutter Tool teamOwned by Flutter Tool teamtoolAffects the "flutter" command-line tool. See also t: labels.Affects the "flutter" command-line tool. See also t: labels.tool-still-validIssues that have been deemed still valid as part of the Flutter Tools issue cleanup initiative.Issues that have been deemed still valid as part of the Flutter Tools issue cleanup initiative.triaged-toolTriaged by Flutter Tool teamTriaged by Flutter Tool team
Activity
gspencergoog commentedon Jun 25, 2021
In #86793, I marked a lot of tests with
no-shuffle. Here are the tests from that PR that need fixing, with a random seed that caused them to fail at the time:Swiftaxe commentedon Aug 6, 2021
@gspencergoog I have started working on this, and fixed a few failing tests. I cannot reproduce some of the failing tests though. For example, all the the tests in the cupertino folder passes successfully for me even when shuffling is turned on.
I remove the 'no-shuffle' tag at the top of the test file and run the chosen test file with --test-randomize-ordering-seed=123 flag. Are you sure the seed you supplied is correct, or am I doing something wrong? 🙂
gspencergoog commentedon Aug 6, 2021
Thanks for looking at these @Swiftaxe! All I know is that some cupertino tests fail for me at that seed (I just tried it again). I'd try running it 100 times with the seed set to "random" and see if it ever fails. It can also be that things fail because of other reasons (flaky tests, timing issues), so perhaps the failure I saw wasn't because of the seed.
My method for for running the tests is:
allow_test_randomization: falseline in thedart_test.yamlinpackages/flutter.flutter test --test-randomize-ordering-seed=123in thepackages/flutterdirectory. I run all the tests, since that's how they'll be run on the server (and we're trying to fix interdependencies here).The failures I currently get in cupertino are:
Swiftaxe commentedon Aug 7, 2021
Thanks for replying @gspencergoog.
I am an experienced developer but new to flutter and open source in general. Is it possible for me to create pull requests for some of the tests at a time?
Dividing the whole lot into several pull requests will avoid one large pull request, and will get things moving faster.
I didn't find any information on this so I don't know how the procedure goes. I'm guessing I need to create new issues related to this one for each pull request, is that right?
gspencergoog commentedon Aug 9, 2021
Sure, in fact that is preferred. That's why I set it up with the tags, so that we could tackle them one at a time instead of all at once.
You don't need to create a new issue for each one, you can just reference this issue in each PR.
You might also read our Contributor's Guide if you want to learn more about how to contribute. Specifically, the Tree Hygiene page.
Swiftaxe commentedon Aug 14, 2021
Thanks! I have finished a bunch and will create a PR.
Animation controller test (#88251)
Animation controller test (flutter#88251)
Fixed leak and removed no-shuffle tag in long_press_test.dart (#88373)
Fixed leak and removed no-shuffle tag in image_stream_test.dart (#88376)
Fixed leak and removed no-shuffle tag in scheduler_test.dart (#88423)
Fixed leak and removed no-shuffle tag in widgets/app_overrides_test.d…
36 remaining items
[material_ui] Remove no shuffle from text field tests (#12547)
Remove no-shuffle from progress indicator test (flutter#186655)
Remove no-shuffle from scrollbar test (flutter#186656)
Remove no-shuffle tag from dropdown_test.dart (flutter#186398)
[material_ui] Remove no-shuffle from progress indicator test (flutter…
[material_ui] Remove no shuffle from text field tests (flutter#12547)
[material_ui] Remove no-shuffle from progress indicator test (flutter…
[material_ui] Remove no shuffle from text field tests (flutter#12547)
Remove no-shuffle from gen_defaults_test (flutter#192280)
[framework] Remove no-shuffle tag definitions and remaining usages (f…
Remove no-shuffle from text_field_test.dart (flutter#193550)
github-actions commentedon Oct 27, 2026
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.[framework] Remove no-shuffle tag definitions and remaining usages (f…