There are a few possibilities to why flutter/engine post-submit is still broken by changes that should have come up in pre:

1. This is garden-variety flakiness, and is being misattributed to pull requests

   UNLIKELY: In the failure in question, there was a both a clear, and intentional change in digest output.

   Click to expand details

   

   

   ... to:

   

   TODO: Nothing. I can't see any work to be done here.

2. The test runner (in this case scenario_app) is not uploading the images properly or timely to Skia Gold

   POSSIBLE, LEANING UNLIKELY: There is fairly extensive logging proving the output of the runner.

   It is possible that despite generating the image, the actual upload is either failing, or not completing in time.

   TODO:

   • Add more logging to scenario_app verifying that addImg calls are succeeding and uploading the image (done: Add more explicit logging (just to stderr) if a try-job detects an untriaged image engine#51454).
   • Modify <SkiaGoldClient>.addImg to report if "Untriaged image" is coming up in pre-submit (log only, no failure) (done: Log messages when an image successfully uploads to Skia Gold engine#51455).

3. The client wrapper that uploads/tests codes in pre-submit unintentionally has diverged from post-submit or flutter/flutter

   POSSIBLE: I need to better understand the command-line arguments to gold_ctl.

   flutter/engine has a fork of SkiaGoldClient, and our implementation is different than the one in flutter/flutter.

   It's possible that either support improved (but was not improved in our forked copy), mistakes were made etc.

   TODO:

   • Review and document the command-line arguments to gold_ctl
   • Review the implementation details of the flutter/flutter copy and flutter/engine fork and look for discrepancies

4. ✅ Flutter Gold (the GitHub check) is either misbehaving (has a bug), or the server API result is stale/not fresh enough

   POSSIBLE: I know very little about this API or the integration.

   TODO:

   • Add logging for the "Flutter Gold" check that sticks around in the commit history/is more easily audit-able.

Background

In the latest example, the flutter/engine tree turned 🔴 due to an untriaged golden file:

The particular rendering issue (see below) is due to YUV textures mis-rendering on Android emulators:

In the originating pull request, all tests (including the "Flutter Gold") check were 🟢 . In fact, if you pull up the execution logs, you'll see that the particular test (testCroppedRotatedMediaSurface_bottomLeft on Impeller/Vulkan) produced the image above, and should have triggered an untriaged digest:

/cc @dnfield @jonahwilliams