Skip to content

Conversation

@gnprice
Copy link
Member

@gnprice gnprice commented Mar 30, 2023

Fixes part of #85160.

I have a draft branch which removes no-shuffle tags from all tests in the framework.

This PR collects up the "easy" fixes:

For all of these files, I reran the tests after my changes both with the previously failing seed mentioned in the comment, and with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.

The fixes I've left for separate PRs are those where the fix is more complex, and those where I believe changes in the flutter_test package are appropriate in order to make it easier either to diagnose the problem or to avoid it in the first place.

Pre-launch Checklist

  • I read the Contributor Guide and followed the process outlined there for submitting PRs.
  • I read the Tree Hygiene wiki page, which explains my responsibilities.
  • I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
  • I signed the CLA.
  • I listed at least one issue that this PR fixes in the description above.
  • I updated/added relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or this PR is test-exempt.
  • All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

@flutter-dashboard flutter-dashboard bot added f: cupertino flutter/packages/flutter/cupertino repository framework flutter/packages/flutter repository. See also f: labels. labels Mar 30, 2023
Copy link
Member

@goderbauer goderbauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you sooo much!

LGTM! LGTM! LGTM!

Copy link
Contributor

@gspencergoog gspencergoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32384589-a60f0e74-c078-11e7-9bc1-e5b5287aea9d

THANK YOU!

@gnprice
Copy link
Member Author

gnprice commented Mar 30, 2023

Thanks!

For all of these files, I reran the tests after my changes […] with 300 random seeds, and they passed every time. That's enough to detect a 1% flake rate with p<.05, and a 2% flake rate with p<.003.

Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit. So I did additional test runs overnight: 500 of them, for a total of 800, all of which passed.

The largest number of tests any of these files have is 104, so if any of them were to have an ordering-dependent flake it's unlikely it would be at a rate lower than 1/104 = 0.96%. (That'd be the probability of a given test coming first, or of coming immediately after another given test.) Running on 800 seeds is enough data to reject that at p<.0004.

@gnprice gnprice added the autosubmit Merge PR when tree becomes green via auto submit App label Mar 30, 2023
@auto-submit auto-submit bot merged commit a28aae9 into flutter:master Mar 30, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 30, 2023
@gnprice gnprice deleted the pr-shuffle-part1 branch March 30, 2023 18:43
@gspencergoog
Copy link
Contributor

Because for some of these test files I couldn't reproduce the problem and don't know what fixed it, I was a bit concerned that one or more of them might just have a low-frequency flake I didn't happen to hit.

Thanks for verifying that. Even if there still is an interdependency or two left in there, it's at a low enough likelihood that we can just deal with it if/when it comes up, which would also expose a seed at which it happens that would let us debug it.

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 31, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Apr 1, 2023
exaby73 pushed a commit to NevercodeHQ/flutter that referenced this pull request Apr 17, 2023
…23751)

Remove no-shuffle from framework tests, part 1: easy cases
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 10, 2023
engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autosubmit Merge PR when tree becomes green via auto submit App f: cupertino flutter/packages/flutter/cupertino repository framework flutter/packages/flutter repository. See also f: labels.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants