Skip to content

customer_testing doesn't update for merges #97595

@stuartmorgan-g

Description

@stuartmorgan-g

The customer_testing test uses a version of flutter/test determined by the initial branch point of the PR, found via https://github.com/flutter/flutter/blob/master/dev/tools/bin/find_commit.dart:

# Next we need to update the flutter/tests checkout.
#
# We use find_commit.dart so that we pull the version of flutter/tests
# that was contemporary when the branch we are on was created. That
# way, we can still run the tests on long-lived branches without being
# affected by breaking changes on trunk causing changes to the tests
# that wouldn't work on the long-lived branch.
#
# (This also prevents trunk from suddenly failing when tests are
# revved on flutter/tests -- if you rerun a passing customer_tests
# shard, it should still pass, even if we rolled one of the tests.)
rm -rf ../../bin/cache/pkg/tests
git clone https://github.com/flutter/tests.git ../../bin/cache/pkg/tests
git -C ../../bin/cache/pkg/tests checkout `dart --enable-asserts ../tools/bin/find_commit.dart ../../bin/cache/pkg/tests`

This causes problems in PRs for people who use a merge workflow to get new versions of master (which is the only one supported by the GitHub UI); the merge will make the PR up to date for the purposes of every test except for customer_testing, creating exactly the kind of mismatch that the script is intended to avoid. The only way to fix a PR in that state is to rebase (from a local checkout) and force-push, which is much less convenient. It's also a very weird failure state; the only way to know this is to have it happen to you, be confused, and ask, which is a frustrating experience for developers.

If possible, when there has been a merge from the original branch find_commit should use the last merged commit instead of the original branch commit to determine the anchor date.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work lista: tests"flutter test", flutter_test, or one of our testsc: contributor-productivityTeam-specific productivity, code health, technical debt.team-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions