Skip to content

Commit 3c12de8

Browse files
authored
android_hardware_smoke_test: Detect blank image failures or EGL initialization warnings and retry (flutter#190110)
This PR addresses flakiness for the android_hardware_smoke_test suite. Occasionally, due to race conditions during rendering composition, platform view tests will produce blank images. The screenshot occurs before composition finished, even though we try to sync it deterministically. For these cases, we introduce a retry right there in the process. Also, we refactor the blank image detection and the existing image cropping functions into image_utils.dart, and add a unit test for the blank image method. Occasionally, for reasons I am not totally clear on, CI starts the test suite in a way which contains EGL initialization errors. This also produces blank images, but retrying wouldn't help at all because it's not a timing issue, it's a setup issue. So we introduce a detection mechanism for these which looks for errors in the logcat, then retries by restarting the activity completely, which at least has a theoretical chance to start up again without the EGL initialization problem. These issues are affecting prod and staging for the instrumented shards, though only for the vulkan tests. OpenGLES tests are passing consistently. I don't see any failures of this type in presubmit, though it may be masked by a different problem related to infrastructure failures during dependency downloads. This is intended to improve flutter#189079 and flutter#189843 (comment) ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant documentation (doc comments with `///`). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 90fcfdf commit 3c12de8

8 files changed

Lines changed: 497 additions & 101 deletions

File tree

‎dev/bots/suite_runners/run_android_hardware_smoke_tests.dart‎

Lines changed: 98 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
// found in the LICENSE file.
44

55
import 'dart:async';
6-
import 'dart:io' show Platform;
6+
import 'dart:io' as io;
77

88
import 'package:file/file.dart';
99
import 'package:file/local.dart';
@@ -36,13 +36,7 @@ Future<void> runAndroidHardwareSmokeTests({
3636

3737
final String testDir = path.join('dev', 'integration_tests', 'android_hardware_smoke_test');
3838

39-
// Regenerate standard Android Gradle wrappers
40-
await runCommand('flutter', <String>[
41-
'create',
42-
'--platform=android',
43-
'--no-overwrite',
44-
'.',
45-
], workingDirectory: testDir);
39+
await _regenerateAndroidWrappers(testDir);
4640

4741
final String androidDir = path.join(testDir, 'android');
4842
final File androidManifestXml = const LocalFileSystem().file(
@@ -55,35 +49,20 @@ Future<void> runAndroidHardwareSmokeTests({
5549
);
5650

5751
try {
58-
// Replace whatever the current backend is with the specified backend.
59-
final impellerBackendMetadata = RegExp(_impellerBackendMetadata(value: '[^"]*'));
60-
if (!impellerBackendMetadata.hasMatch(androidManifestContents)) {
61-
throw StateError(
62-
'Could not find io.flutter.embedding.android.ImpellerBackend meta-data tag inside AndroidManifest.xml',
63-
);
64-
}
65-
androidManifestXml.writeAsStringSync(
66-
androidManifestContents.replaceFirst(
67-
impellerBackendMetadata,
68-
_impellerBackendMetadata(value: backend.name),
69-
),
70-
);
52+
_setAndroidManifestBackend(androidManifestXml, androidManifestContents, backend);
7153

72-
// 1. Run driver tests to generate reference screenshots
73-
await runCommand('flutter', <String>[
74-
'drive',
75-
'--driver=test_driver/driver_test.dart',
76-
'--target=integration_test/integration_test_wrapper.dart',
77-
'--no-dds',
78-
'--no-enable-dart-profiling',
79-
], workingDirectory: testDir);
54+
// 1. Run driver tests to generate reference screenshots (with retry loop)
55+
final bool success = await _runRetryLoop(testDir);
56+
if (!success) {
57+
return;
58+
}
8059

8160
if (runInstrumented) {
8261
final String gradle = path.absolute(
83-
path.join(androidDir, Platform.isWindows ? 'gradlew.bat' : 'gradlew'),
62+
path.join(androidDir, io.Platform.isWindows ? 'gradlew.bat' : 'gradlew'),
8463
);
8564

86-
// 3. Build and run the instrumented tests.
65+
// 2. Build and run the instrumented tests.
8766
await runCommand(gradle, <String>[
8867
':app:connectedDebugAndroidTest',
8968
'-Pandroid.testInstrumentationRunnerArguments.class=com.example.android_hardware_smoke_test.FlutterActivityTest',
@@ -98,3 +77,91 @@ Future<void> runAndroidHardwareSmokeTests({
9877
_cleanGoldensDirectory(destinationDir);
9978
}
10079
}
80+
81+
Future<void> _regenerateAndroidWrappers(String testDir) async {
82+
await runCommand('flutter', <String>[
83+
'create',
84+
'--platform=android',
85+
'--no-overwrite',
86+
'.',
87+
], workingDirectory: testDir);
88+
}
89+
90+
void _setAndroidManifestBackend(File file, String contents, ImpellerBackend backend) {
91+
final impellerBackendMetadata = RegExp(_impellerBackendMetadata(value: '[^"]*'));
92+
if (!impellerBackendMetadata.hasMatch(contents)) {
93+
throw StateError(
94+
'Could not find io.flutter.embedding.android.ImpellerBackend meta-data tag inside AndroidManifest.xml',
95+
);
96+
}
97+
file.writeAsStringSync(
98+
contents.replaceFirst(impellerBackendMetadata, _impellerBackendMetadata(value: backend.name)),
99+
);
100+
}
101+
102+
Future<bool> _runRetryLoop(String testDir) async {
103+
const maxAttempts = 3;
104+
var exitCode = 0;
105+
106+
for (var attempt = 1; attempt <= maxAttempts; attempt++) {
107+
// Clear logcat buffer before running
108+
await runCommand('adb', <String>['logcat', '-c']);
109+
110+
final driveArgs = <String>[
111+
'drive',
112+
'--driver=test_driver/driver_test.dart',
113+
'--target=integration_test/integration_test_wrapper.dart',
114+
'--no-dds',
115+
'--no-enable-dart-profiling',
116+
];
117+
118+
if (attempt > 1) {
119+
driveArgs.add('--no-build');
120+
}
121+
122+
final Command command = await startCommand('flutter', driveArgs, workingDirectory: testDir);
123+
exitCode = await command.process.exitCode;
124+
if (exitCode == 0) {
125+
return true;
126+
}
127+
128+
io.stderr.writeln(
129+
'flutter drive failed with exit code $exitCode on attempt $attempt/$maxAttempts.',
130+
);
131+
132+
// Inspect the process logcat on failure to detect if a transient EGL/graphics context
133+
// negotiation error occurred during startup, enabling a safe activity/process level retry.
134+
final bool hasEglWarning = await _checkForTransientEglFailure();
135+
if (!hasEglWarning) {
136+
// Non-retryable error: exit immediately and log specific failure
137+
foundError(<String>[
138+
'Android Hardware Smoke Tests driver run failed with exit code $exitCode and no transient EGL warning was found in logcat.',
139+
]);
140+
return false;
141+
}
142+
143+
// Retryable EGL warning: log progress and continue if attempts remain
144+
if (attempt < maxAttempts) {
145+
io.stderr.writeln(
146+
'attempt $attempt of $maxAttempts: detected retryable EGL initialization warning. Retrying...',
147+
);
148+
}
149+
}
150+
151+
// Loop finished: exhausted all attempts
152+
foundError(<String>[
153+
'Android Hardware Smoke Tests driver run failed to initialize EGL after $maxAttempts attempts.',
154+
]);
155+
return false;
156+
}
157+
158+
Future<bool> _checkForTransientEglFailure() async {
159+
try {
160+
final String logcatOutput = await runAndGetStdout('adb', <String>['logcat', '-d']).join('\n');
161+
return logcatOutput.contains('Failed to choose config with EGL_SWAP_BEHAVIOR_PRESERVED') ||
162+
logcatOutput.contains('Failed to initialize 101010-2 format');
163+
} catch (e) {
164+
io.stderr.writeln('Warning: Failed to check logcat for EGL failure: $e');
165+
return false;
166+
}
167+
}

‎dev/integration_tests/android_hardware_smoke_test/README.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -479,3 +479,38 @@ a single step, the test suite executes exactly once,
479479
remains low maintenance across AGP upgrades, and
480480
cleanly bubbles up any legitimate compiler or runner
481481
errors without silent try-catch blocks.
482+
483+
---
484+
485+
## 7. Graphics Initialization & Screenshot Retry Mechanisms
486+
487+
To guarantee high stability against flakiness on emulators and physical hardware, the test suite implements a two-layered retry mechanism:
488+
489+
### A. Process & Activity-Level Retries (Graphics Context Initialization)
490+
* **Goal**: Recovers from transient EGL or graphics context negotiation errors (e.g., `Failed to choose config with EGL_SWAP_BEHAVIOR_PRESERVED` or HWUI startup warnings) that occur globally when the Flutter Engine boots up.
491+
* **Mechanism**:
492+
* **On Host (CI)**: If the `flutter drive` run fails, the host suite runner
493+
([`run_android_hardware_smoke_tests.dart`](dev/bots/suite_runners/run_android_hardware_smoke_tests.dart))
494+
checks the device logcat. If transient EGL errors are detected, it restarts
495+
the test run process (up to 3 attempts) to initialize a new EGL context in
496+
a fresh process.
497+
* **On Device (JUnit)**: If an attempt fails, the JUnit runner
498+
([`FlutterActivityTest.kt`](android/app/src/androidTest/java/com/example/android_hardware_smoke_test/FlutterActivityTest.kt))
499+
recreates the Activity scenario and retries **only** on blank screenshot
500+
failures (up to 3 attempts). It does **not** retry on EGL/graphics errors,
501+
as recreating the activity within the same application process cannot
502+
recover from process-level EGL initialization failures; these fail
503+
immediately on the device to bubble up to the host runner. Standard test
504+
failures (e.g., golden pixel mismatches) also fail immediately.
505+
506+
### B. Screenshot-Level Retries (Blank Platform View Capture)
507+
* **Goal**: Prevents race conditions where a screenshot is taken before native platform view compositing has fully settled, resulting in a blank/transparent frame.
508+
* **Mechanism**:
509+
* **Both Modes**: The screenshot capture loop (both native `UiAutomation` and
510+
host-side `NativeDriver`) checks the cropped image. If the cropped image is
511+
completely transparent or solid black, it retries the capture up to 3
512+
times, sleeping 200ms between attempts.
513+
* **Short-circuiting**: If an EGL/graphics error is detected in the logcat
514+
during screenshot attempts, the loop is terminated early (bypassing
515+
remaining screenshot retries) to fail the test immediately and bubble the
516+
failure up to the host runner as quickly as possible.

0 commit comments

Comments
 (0)