Report failed Skia golden comparisons on post-submit should as test failures #162621
Description
In #162400, I attempted to implement (local to a test) retry logic for failing Skia golden comparisons:
int retriesLeft = 2;
do {
try {
await expectLater(
nativeDriver.screenshot(),
matchesGoldenFile('$goldenPrefix.blue_orange_gradient_portrait.png'),
);
break;
} on TestFailure catch (e) {
if (retriesLeft == 0) {
rethrow;
}
print('Caught: $e. Retrying...');
retriesLeft--;
}
} while (retriesLeft > 0);Unfortunately, it does not work; matchesGoldenFile never receives a TestFailure, which is what is expected:
flutter/packages/flutter_test/lib/src/_matchers_io.dart
Lines 77 to 83 in df114fb
As a result, an unhandled exception is thrown instead of the matcher reporting a test failure.
You can see an example of the failure here:
00:01 �[32m+0�[0m: should screenshot and match a blue -> orange gradient�[0m
Updating golden file: android_engine_test.vulkan.hybrid_composition_platform_view.blue_orange_gradient_portrait.png (12504 bytes)...
00:05 �[32m+0�[0m�[31m -1�[0m: should screenshot and match a blue -> orange gradient �[1m�[31m[E]�[0m�[0m
SkiaException: Skia Gold received an unapproved image in post-submit
testing. Golden file images in flutter/flutter are triaged
in pre-submit during code review for the given PR.
Visit https://flutter-gold.skia.org// to view and approve
the image(s), or revert the associated change. For more
information, visit the wiki:
https://github.com/flutter/flutter/blob/main/docs/contributing/testing/Writing-a-golden-file-test-for-package-flutter.md
Debug information for Gold --------------------------------
stdout: Given image with hash 5ba6df7acef516f7dab78587cf30ce20 for test android_engine_test.vulkan.hybrid_composition_platform_view.blue_orange_gradient_portrait
Expectation for test: 323c776b267f0a96428abea72a1e4a6b (positive)
Untriaged or negative image: https://flutter-gold.skia.org/detail?grouping=name%3Dandroid_engine_test.vulkan.hybrid_composition_platform_view.blue_orange_gradient_portrait%26source_type%3Dflutter&digest=5ba6df7acef516f7dab78587cf30ce20
stderr: Test: android_engine_test.vulkan.hybrid_composition_platform_view.blue_orange_gradient_portrait FAIL
result-state.json: No result file found.
package:flutter_goldens/skia_client.dart 261:7 SkiaGoldClient.imgtestAdd
===== asynchronous gap ===========================
package:android_driver_extensions/src/goldens.dart 229:28 _MatchesGoldenFile.matchAsync
===== asynchronous gap ===========================
test_driver/platform_view/hybrid_composition_platform_view_main_test.dart 55:9 main.<fn>