-
Notifications
You must be signed in to change notification settings - Fork 29.7k
Description
The customer_testing test uses a version of flutter/test determined by the initial branch point of the PR, found via https://github.com/flutter/flutter/blob/master/dev/tools/bin/find_commit.dart:
flutter/dev/customer_testing/ci.sh
Lines 21 to 34 in 72e426f
| # Next we need to update the flutter/tests checkout. | |
| # | |
| # We use find_commit.dart so that we pull the version of flutter/tests | |
| # that was contemporary when the branch we are on was created. That | |
| # way, we can still run the tests on long-lived branches without being | |
| # affected by breaking changes on trunk causing changes to the tests | |
| # that wouldn't work on the long-lived branch. | |
| # | |
| # (This also prevents trunk from suddenly failing when tests are | |
| # revved on flutter/tests -- if you rerun a passing customer_tests | |
| # shard, it should still pass, even if we rolled one of the tests.) | |
| rm -rf ../../bin/cache/pkg/tests | |
| git clone https://github.com/flutter/tests.git ../../bin/cache/pkg/tests | |
| git -C ../../bin/cache/pkg/tests checkout `dart --enable-asserts ../tools/bin/find_commit.dart ../../bin/cache/pkg/tests` |
This causes problems in PRs for people who use a merge workflow to get new versions of master (which is the only one supported by the GitHub UI); the merge will make the PR up to date for the purposes of every test except for customer_testing, creating exactly the kind of mismatch that the script is intended to avoid. The only way to fix a PR in that state is to rebase (from a local checkout) and force-push, which is much less convenient. It's also a very weird failure state; the only way to know this is to have it happen to you, be confused, and ask, which is a frustrating experience for developers.
If possible, when there has been a merge from the original branch find_commit should use the last merged commit instead of the original branch commit to determine the anchor date.