Skip to content

Commit 1f33e88

Browse files
authored
Fix android_hardware_smoke_test retry mechanisms (flutter#195708)
Fixes flutter#194928 Fixes flutter#191348 Fixes flutter#193557 Related to flutter#193205 This makes `android_hardware_smoke_test` recover cleanly from transient emulator `SurfaceFlinger` wedges and host-GPU-pipe startup crashes instead of timing out or failing the shard after a passing retry. See flutter#193557 (comment) for my root-cause investigation and local emulator reproduction. ## What changed `run_android_hardware_smoke_tests.dart` already retried failed `flutter drive` runs once, while invoking `adb logcat -c` and `flutter drive` through `runCommand`, which called `foundError` on any non-zero exit and marked the shard failed even when attempt 2 passed. `runCommand` now supports `checkExitCode: false` so retryable commands do not record a shard error on attempt 1, and `startCommand` decodes output streams with `Utf8Decoder(allowMalformed: true)` so raw bytes in native crash tombstones do not throw `FormatException` when inspecting `adb logcat -d`. Retrying with `flutter drive --no-build` only restarted the app process and left `SurfaceFlinger` wedged when an emulator QuickBoot snapshot broke display composition. When attempt 1 fails with a blank screenshot or fatal graphics error in `adb logcat -d`, the runner now resets `sys.boot_completed` to `0`, restarts `surfaceflinger` and `zygote` via `adb shell stop` and `adb shell start`, waits for `sys.boot_completed=1`, and unlocks the screen before attempt 2. Normal test assertion failures and golden mismatches still fail immediately on attempt 1. When the app crashed during native EGL or Vulkan startup right after printing its Dart VM service URI, `flutter drive` looped against the closed `adb forward` port until LUCI killed the shard on timeout. `startCommand` and `runCommand` now accept a `stderrListener` alongside `outputListener` so the runner can watch `stdout` and `stderr` for slow-VM-connect warnings, check `adb shell pidof`, send `SIGKILL` to `flutter drive` once the app process has exited, and trigger the userspace-restart retry immediately. In `FlutterActivityTest.kt`, querying `logcat -d --pid=$pid *:W` stripped `Log.i` attempt markers, while both `FlutterActivityTest.kt` and `run_android_hardware_smoke_tests.dart` treated routine `HWUI` warnings as fatal errors. `FlutterActivityTest.kt` now passes `$TAG:I *:W` to keep attempt markers visible in the dump. Both files now match explicit fatal EGL, Vulkan, and host-pipe errors while ignoring benign `HWUI` capability-probe logs. ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 52bfbdd commit 1f33e88

7 files changed

Lines changed: 588 additions & 172 deletions

File tree

‎dev/bots/run_command.dart‎

Lines changed: 54 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -79,12 +79,18 @@ class CommandResult {
7979
/// running process.
8080
///
8181
/// `outputListener` is called for every line of standard output from the
82-
/// process, and is given the [io.Process] object. This can be used to interrupt
83-
/// an indefinitely running process, for example, by waiting until the process
84-
/// emits certain output.
82+
/// process, and `stderrListener` is called for every line of standard error.
83+
/// Both are given the [io.Process] object, which can be used to interrupt an
84+
/// indefinitely running process (for example, after waiting until the process
85+
/// emits certain output).
8586
///
86-
/// `outputMode` controls where the standard output from the command process
87-
/// goes. See [OutputMode].
87+
/// Standard output and standard error are decoded as UTF-8 with
88+
/// `allowMalformed: true` so non-UTF-8 bytes (such as raw crash tombstones in
89+
/// `adb logcat`) are replaced with U+FFFD rather than throwing a
90+
/// [FormatException].
91+
///
92+
/// `outputMode` controls where the standard output and standard error from the
93+
/// command process go. See [OutputMode].
8894
Future<Command> startCommand(
8995
String executable,
9096
List<String> arguments, {
@@ -93,6 +99,7 @@ Future<Command> startCommand(
9399
OutputMode outputMode = OutputMode.print,
94100
bool Function(String)? removeLine,
95101
void Function(String, io.Process)? outputListener,
102+
void Function(String, io.Process)? stderrListener,
96103
}) async {
97104
final String relativeWorkingDir = path.relative(workingDirectory ?? io.Directory.current.path);
98105
final commandDescription =
@@ -118,7 +125,7 @@ Future<Command> startCommand(
118125
process,
119126
time,
120127
process.stdout
121-
.transform<String>(const Utf8Decoder())
128+
.transform<String>(const Utf8Decoder(allowMalformed: true))
122129
.transform(const LineSplitter())
123130
.where((String line) => removeLine == null || !removeLine(line))
124131
.map<String>((String line) {
@@ -136,9 +143,13 @@ Future<Command> startCommand(
136143
})
137144
.join('\n'),
138145
process.stderr
139-
.transform<String>(const Utf8Decoder())
146+
.transform<String>(const Utf8Decoder(allowMalformed: true))
140147
.transform(const LineSplitter())
141148
.map<String>((String line) {
149+
final formattedLine = '$line\n';
150+
if (stderrListener != null) {
151+
stderrListener(formattedLine, process);
152+
}
142153
switch (outputMode) {
143154
case OutputMode.print:
144155
print(line);
@@ -223,30 +234,50 @@ String? _discoverBestNdkPath() {
223234

224235
/// Runs the `executable` and waits until the process exits.
225236
///
226-
/// If the process exits with a non-zero exit code and `expectNonZeroExit` is
227-
/// false, calls foundError (which does not terminate execution!).
237+
/// Exit code validation is controlled by `checkExitCode`, `expectNonZeroExit`,
238+
/// and `expectedExitCode`:
239+
/// * When `checkExitCode` is true (the default), unexpected exit codes record a
240+
/// shard failure via [foundError] (which does not terminate execution):
241+
/// - By default (`expectNonZeroExit` is false and `expectedExitCode` is null),
242+
/// any non-zero exit code fails.
243+
/// - If `expectNonZeroExit` is true, exit code `0` fails and any non-zero exit
244+
/// code succeeds (unless `expectedExitCode` is also set to require a
245+
/// specific non-zero exit code).
246+
/// * When `checkExitCode` is false, exit code validation is skipped and
247+
/// [foundError] is never called, allowing callers (such as retry loops or
248+
/// readiness probes) to inspect [CommandResult.exitCode] directly. Both
249+
/// `expectNonZeroExit` and `expectedExitCode` must be left at their defaults
250+
/// when `checkExitCode` is false.
228251
///
229252
/// `outputListener` is called for every line of standard output from the
230-
/// process, and is given the [io.Process] object. This can be used to interrupt
231-
/// an indefinitely running process, for example, by waiting until the process
232-
/// emits certain output.
253+
/// process, and `stderrListener` is called for every line of standard error.
254+
/// Both are given the [io.Process] object, which can be used to interrupt an
255+
/// indefinitely running process (for example, after waiting until the process
256+
/// emits certain output).
233257
///
234258
/// Returns the result of the finished process.
235259
///
236-
/// `outputMode` controls where the standard output from the command process
237-
/// goes. See [OutputMode].
260+
/// `outputMode` controls where the standard output and standard error from the
261+
/// command process go. See [OutputMode].
262+
// TODO(andywolff): Consolidate checkExitCode, expectNonZeroExit, and expectedExitCode, https://github.com/flutter/flutter/issues/196126
238263
Future<CommandResult> runCommand(
239264
String executable,
240265
List<String> arguments, {
241266
String? workingDirectory,
242267
Map<String, String>? environment,
268+
bool checkExitCode = true,
243269
bool expectNonZeroExit = false,
244270
int? expectedExitCode,
245271
String? failureMessage,
246272
OutputMode outputMode = OutputMode.print,
247273
bool Function(String)? removeLine,
248274
void Function(String, io.Process)? outputListener,
275+
void Function(String, io.Process)? stderrListener,
249276
}) async {
277+
assert(
278+
checkExitCode || (!expectNonZeroExit && expectedExitCode == null),
279+
'expectNonZeroExit and expectedExitCode cannot be set when checkExitCode is false.',
280+
);
250281
final commandDescription =
251282
'${path.relative(executable, from: workingDirectory)} ${arguments.join(' ')}';
252283
final String relativeWorkingDir = workingDirectory ?? path.relative(io.Directory.current.path);
@@ -268,6 +299,7 @@ Future<CommandResult> runCommand(
268299
outputMode: outputMode,
269300
removeLine: removeLine,
270301
outputListener: outputListener,
302+
stderrListener: stderrListener,
271303
);
272304

273305
final result = CommandResult._(
@@ -277,8 +309,9 @@ Future<CommandResult> runCommand(
277309
await command._savedStderr,
278310
);
279311

280-
if ((result.exitCode == 0) == expectNonZeroExit ||
281-
(expectedExitCode != null && result.exitCode != expectedExitCode)) {
312+
if (checkExitCode &&
313+
((result.exitCode == 0) == expectNonZeroExit ||
314+
(expectedExitCode != null && result.exitCode != expectedExitCode))) {
282315
// Print the output when we get unexpected results (unless output was
283316
// printed already).
284317
switch (outputMode) {
@@ -327,18 +360,12 @@ String _prettyPrintRunCommand(String executable, List<String> arguments, String?
327360

328361
/// Specifies what to do with the command output from [runCommand] and [startCommand].
329362
enum OutputMode {
330-
/// Forwards standard output and standard error streams to the test process'
331-
/// standard output stream (i.e. stderr is redirected to stdout).
332-
///
333-
/// Use this mode if all you want is print the output of the command to the
334-
/// console. The output is no longer available after the process exits.
363+
/// Forwards standard output and standard error lines to [print] while also
364+
/// saving them on the returned [CommandResult].
335365
print,
336366

337-
/// Saves standard output and standard error streams in memory.
338-
///
339-
/// Captured output can be retrieved from the [CommandResult] object.
340-
///
341-
/// Use this mode in tests that need to inspect the output of a command, or
342-
/// when the output should not be printed to console.
367+
/// Saves standard output and standard error in memory on the returned
368+
/// [CommandResult] without forwarding lines to [print] (unless [runCommand]
369+
/// records an unexpected exit when `checkExitCode` is true).
343370
capture,
344371
}

0 commit comments

Comments
 (0)