This pull request was opened from and to a release candidate branch. This should only be done as part of the official Flutter release process. If you are attempting to make a regular contribution to the Flutter project, please close this PR and follow the instructions at Tree Hygiene for detailed instructions on contributing to Flutter.

Reviewers: Use caution before merging pull requests to release branches. Ensure the proper procedure has been followed.

The first failure didn't have the GOLDCTL set and the PR that updated the engine version didn't run skia gold like it should have. I know ideally this would have worked the first try, but we really should try to get it working. Testing shouldn't really be optional. I'm happy to meet up on monday to try to connect the dots.

@gaaclarke this is a release branch, Skia gold doesn't work on it - and we shouldn't be blocking the beta on trying to make it work.

Testing shouldn't really be optional. I'm happy to meet up on monday to try to connect the dots.

To add some further context, I noticed this workflow did not appear to work a couple of months ago, but the release team told me not to worry about it, and de-flake. On the release branch I was testing, however, I was seeing 100% failure rate.

@gaaclarke, I know you did test this on a specific branch, but is it possible that it wasn't comparing the right goldens, and the generated goldens just happened to match the main branch ones? I am skeptical that this release infrastructure ever worked, and thus I don't think we should block the release on implementing this new release infrastructure correctly for the first time.

cc @matanlurey

@gaaclarke, I know you did test this on a specific branch, but is it possible that it wasn't comparing the right goldens, and the generated goldens just happened to match the main branch ones?

Nope, that's not how it works. Here was the trial run: #52005

The way it should work is that the PR that sets the release version will generate a whole new branch of goldens (notice how all the generated goldens have the release appended to their name). That's how we knew it worked. That did not seem to happen in your PR that set the release number though. Which if it doesn't, nothing is going to work.

Sorry I an just getting back to the swing of things. Let's do this:

1. @christopherfujino should have the authority/decision making power to do what is necessary to get the release out. I don't know if there is a sensitive timeline that requires this change over figuring out why it's not working as intended, but I also defer that to Chris. tl;dr: Whether to skip the tests is a Chris decision.
2. I am very confused at @jonahwilliams's comment of:

this is a release branch, Skia gold doesn't work on it

That just .. doesn't make sense to me, we spent significant time designing and testing doing just that:

https://github.com/flutter/engine/tree/main/testing/skia_gold_client#release-testing

Yeah I was out of the loop a bit, we did actually make that work.

I chatted with Aaron on this offline, in summary:

1. It's unclear to us, but one of the following is true:
   a. This worked at one point when Aaron verified it on his test branch, but was later regressed; or
   b. This only worked for a specific set of circumstances (such that his test branch worked), but not in a robust way for all releases, which is why these tests have been seen to fail on multiple RC branches since
2. I highly suspect what is wrong here is some misconfiguration of the infrastructure, either in the recipes, builder config, engine .ci.yaml, or *.json builder configs in the engine tree, given that the error is a missing env variable
3. Unfortunately, testing/debugging the release workflow is incredibly difficult, because it runs in a more secure context than normal LUCI builds
4. Given the release team's current priorities, we will skip the test to unblock this beta release
5. If someone on the engine team is able to pinpoint the bug and offer a CP, I will approve it and merge it into this branch for a CP release.
6. I can assist someone by creating branches and possibly granting them temporary access to read LUCI logs, but I don't have the time to more actively work on investigating this.