Ergonomics of re-runs + autosubmit are worse under unified checks #191005

Help us understand the severity of this issue

• breakage; unable to contribute or trigger builds will no workarounds
• inconvenient workarounds exist, but significant effort required
• needed for a Flutter team-wide priority (already agreed upon)
• nice-to-have; none of the above

What do you need help with

The interaction between autosubmit and the new unified check runs is (at least in flutter/packages, but I assume this happens in flutter/flutter as well) much less convenient when re-running flaky tests than the pre-unification flow.

(For context here: flutter/packages runs a lot of tests, across a lot of different types of tests, and there's frequently a non-trivial amount of flake. Obviously that's not ideal, obviously we should not have flake and we do try to keep it under control, but it's a constant battle that we never win so a regression in how it impacts engineers is a problem.)

The old workflow when something that we wanted to autosubmit had presubmit flake was:

• trigger a re-run of the flaky tests
• add autosubmit
• go do other things

Because a re-run immediately changed the status back to pending, autosubmit would wait patiently in the background, and then (assuming it didn't flake again) immediately land the PR once the test completed, with no further interation.

The new workflow is:

• trigger a re-run
• go do other things
• at some point remember that you were trying to land something, and go check on it
• if it's green, add autosubmit

This requires two manual interventions, separated by a bunch of time, and with no notification system. I've had this add a day to landing because I got busy after the second step and forgot to check back until the next day.

This is necessary because the status of the unified check seems to stay failed until after the re-runs complete, rather than becoming pending again.