Scenario app skia-gold does not work, or does not work as expected #144352
Description
STATUS: Known issues are WIP:
- Android scenario app tests with platform views + rotations sometimes display different "gray" color #144657
- Deprecate crop/rotation support for Android plugins, consider Dart alternative #144407
-
ExternalTexturesTeststill sometimes crashes on rendering the Android image #144365 - Flutter doctor should check using the latest emulator #144232
- [scenario app] golden failure due to single pixel dif #145058
While I'm filing this with e: scenario-app it's possible the scope is beyond scenario_app. We are likely "holding it wrong", but it's catastrophic the way it's currently configured - we either turn the tree red because we didn't actually authorize all the digests (example 1), or we straight up miss digests that show incorrect behavior (example 2).
Example 1: flutter/engine#51061 merged, and turned the tree red.
Only a single image came up in the PR, which was approved.
When it landed post-submit turned the tree red:
Debug information for Gold --------------------------------
stdout: Given image with hash 31dd46a724903b2ba29d64e579a691f1 for test PlatformTextureUiTests_testPlatformViewRotate
Expectation for test: 668bcdeda2f38520c865e639f535b05e (positive)
Expectation for test: 9eb7578aac2db40027790c7f57a2663f (positive)
Expectation for test: 2d6d812504637cb9638d70595733fa0c (positive)
Expectation for test: bbd9733e90b1c0fbde2f7c1689e85f48 (positive)
Expectation for test: fbfc2e60b6b93e5e99659d2674867bb3 (positive)
Expectation for test: af46302575b693eb83c40d2bee719017 (positive)
Expectation for test: 33930e542bfc4c1296f3ad1ad6c49cd2 (positive)
Expectation for test: 3022c9919e2bdef020c73bd1acfd7534 (positive)
Expectation for test: 6bd005faf9dc5939147ef2807c987256 (positive)
Expectation for test: 7d873804f96940f215a7c883d3f8fa4e (positive)
Expectation for test: da554cb3ec3966b1f1bd9c6afb229a45 (positive)
Expectation for test: fd830630715ae54de49c1341e17ad852 (positive)
Expectation for test: 2674732f5e64ea5aafcbd6d7e735bf19 (positive)
Fetching most recent positive digest for trace with ID "331c528375c030eeda1a0d4da269182d".
No recent positive digests for trace with ID "331c528375c030eeda1a0d4da269182d". This probably means that the test was newly added.
Non-exact image comparison using algorithm "fuzzy" against most recent positive digest "".
Untriaged or negative image: https://flutter-engine-gold.skia.org/detail?grouping=name%3DPlatformTextureUiTests_testPlatformViewRotate%26source_type%3Dflutter-engine&digest=31dd46a724903b2ba29d64e579a691f1
stderr: Test: PlatformTextureUiTests_testPlatformViewRotate FAILThis should ... just never happen?
Example 2: flutter/engine#50730 generated images do not show up.
In flutter/engine#50730, @chinmaygarde fixed/completed a feature for Impeller Vulkan and external textures. Once again, Skia gold did pop up and ask for triage:
... but the images aren't related to the patch. I instead went looking directly in the logs:
... and pulled out the real golden change, which shows a bug not even shown by Skia gold:
